We have noticed disk quota failures with jobs for the opensource dpdk project since Friday 03/13.
Trying to add some debug like df -h in jobs of mine, I get quite random outputs depending on the environment the job is executed on:
hostname: edb47637-c344-4a03-8538-711046469ad4@14975.lxd-arm64-01-com
hostname: 5f37dcf5-9079-405b-b474-693481472995@25560.lxd-arm64-03-com
hostname: 306788d5-7614-4e9d-856a-1043cb4aeaf4@6138.lxd-arm64-02-com
Is there something changing on arm containers side at the moment?
Is it normal to see those (hw an sw) differences between systems?
1 Like
Latest failure on master branch:https://travis-ci.com/github/DPDK/dpdk/jobs/299673401#L2951
gcc -o examples/dpdk-vmdq_dcb ‘examples/examples@@dpdk-vmdq_dcb @exe/vmdq_dcb_main.c.o’ -Wl,–no-undefined -Wl,–as-needed -Wl,–whole-archive -Wl,–start-group drivers/librte_common_cpt.a drivers/librte_common_dpaax.a drivers/librte_common_iavf.a drivers/librte_common_mlx5.a drivers/librte_common_octeontx.a drivers/librte_common_octeontx2.a drivers/librte_bus_dpaa.a drivers/librte_bus_fslmc.a drivers/librte_bus_ifpga.a drivers/librte_bus_pci.a drivers/librte_bus_vdev.a drivers/librte_bus_vmbus.a drivers/librte_mempool_bucket.a drivers/librte_mempool_dpaa.a drivers/librte_mempool_dpaa2.a drivers/librte_mempool_octeontx.a drivers/librte_mempool_octeontx2.a drivers/librte_mempool_ring.a drivers/librte_mempool_stack.a drivers/librte_pmd_af_packet.a drivers/librte_pmd_ark.a drivers/librte_pmd_atlantic.a drivers/librte_pmd_avp.a drivers/librte_pmd_axgbe.a drivers/librte_pmd_bond.a drivers/librte_pmd_bnx2x.a drivers/librte_pmd_bnxt.a drivers/librte_pmd_cxgbe.a drivers/librte_pmd_dpaa.a drivers/librte_pmd_dpaa2.a drivers/librte_pmd_e1000.a drivers/librte_pmd_ena.a drivers/librte_pmd_enetc.a drivers/librte_pmd_enic.a drivers/librte_pmd_failsafe.a drivers/librte_pmd_fm10k.a drivers/librte_pmd_i40e.a drivers/librte_pmd_hinic.a drivers/librte_pmd_hns3.a drivers/librte_pmd_iavf.a drivers/librte_pmd_ice.a drivers/librte_pmd_ipn3ke.a drivers/librte_pmd_ixgbe.a drivers/librte_pmd_kni.a drivers/librte_pmd_liquidio.a drivers/librte_pmd_memif.a drivers/librte_pmd_mlx4.a drivers/librte_pmd_mlx5.a drivers/librte_pmd_netvsc.a drivers/librte_pmd_nfp.a drivers/librte_pmd_null.a drivers/librte_pmd_octeontx.a drivers/librte_pmd_octeontx2.a drivers/librte_pmd_pcap.a drivers/librte_pmd_pfe.a drivers/librte_pmd_qede.a drivers/librte_pmd_ring.a drivers/librte_pmd_softnic.a drivers/librte_pmd_tap.a drivers/librte_pmd_thunderx.a drivers/librte_pmd_vdev_netvsc.a drivers/librte_pmd_vhost.a drivers/librte_pmd_virtio.a drivers/librte_pmd_vmxnet3.a drivers/librte_rawdev_dpaa2_cmdif.a drivers/librte_rawdev_dpaa2_qdma.a drivers/librte_rawdev_ifpga.a drivers/librte_rawdev_ntb.a drivers/librte_rawdev_octeontx2_dma.a drivers/librte_rawdev_octeontx2_ep.a drivers/librte_rawdev_skeleton.a drivers/librte_pmd_caam_jr.a drivers/librte_pmd_ccp.a drivers/librte_pmd_dpaa_sec.a drivers/librte_pmd_dpaa2_sec.a drivers/librte_pmd_nitrox.a drivers/librte_pmd_null_crypto.a drivers/librte_pmd_octeontx_crypto.a drivers/librte_pmd_octeontx2_crypto.a drivers/librte_pmd_openssl.a drivers/librte_pmd_crypto_scheduler.a drivers/librte_pmd_virtio_crypto.a drivers/librte_pmd_octeontx_compress.a drivers/librte_pmd_qat.a drivers/librte_pmd_zlib.a drivers/librte_pmd_ifc.a drivers/librte_pmd_mlx5_vdpa.a drivers/librte_pmd_dpaa_event.a drivers/librte_pmd_dpaa2_event.a drivers/librte_pmd_octeontx2_event.a drivers/librte_pmd_opdl_event.a drivers/librte_pmd_skeleton_event.a drivers/librte_pmd_sw_event.a drivers/librte_pmd_dsw_event.a drivers/librte_pmd_octeontx_event.a drivers/librte_pmd_bbdev_null.a drivers/librte_pmd_bbdev_turbo_sw.a drivers/librte_pmd_bbdev_fpga_lte_fec.a -Wl,–no-whole-archive -Wl,–no-as-needed -pthread -lm -ldl -lnuma lib/librte_eal.a lib/librte_kvargs.a lib/librte_mempool.a lib/librte_ring.a lib/librte_net.a lib/librte_mbuf.a lib/librte_ethdev.a lib/librte_meter.a lib/librte_cmdline.a lib/librte_pci.a lib/librte_cryptodev.a lib/librte_hash.a lib/librte_eventdev.a lib/librte_timer.a lib/librte_rawdev.a lib/librte_stack.a lib/librte_sched.a lib/librte_ip_frag.a lib/librte_security.a lib/librte_kni.a lib/librte_pipeline.a lib/librte_port.a lib/librte_table.a lib/librte_lpm.a lib/librte_acl.a lib/librte_gso.a lib/librte_vhost.a lib/librte_reorder.a lib/librte_compressdev.a lib/librte_bbdev.a -lpcap /usr/lib/aarch64-linux-gnu/libbsd.so -lmlx5 -libverbs -Wl,–end-group /usr/lib/aarch64-linux-gnu/libz.so -lmlx4 -libverbs -lmlx5 -libverbs -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap /usr/lib/aarch64-linux-gnu/libfdt.so /usr/lib/aarch64-linux-gnu/libcrypto.so -lmlx5 -libverbs
collect2: error: ld returned 1 exit status
[snip]
FAILED: examples/dpdk-l3fwd-acl
gcc -o examples/dpdk-l3fwd-acl ‘examples/examples@@dpdk-l3fwd-acl @exe/l3fwd-acl_main.c.o’ -Wl,–no-undefined -Wl,–as-needed -Wl,–whole-archive -Wl,–start-group drivers/librte_common_cpt.a drivers/librte_common_dpaax.a drivers/librte_common_iavf.a drivers/librte_common_mlx5.a drivers/librte_common_octeontx.a drivers/librte_common_octeontx2.a drivers/librte_bus_dpaa.a drivers/librte_bus_fslmc.a drivers/librte_bus_ifpga.a drivers/librte_bus_pci.a drivers/librte_bus_vdev.a drivers/librte_bus_vmbus.a drivers/librte_mempool_bucket.a drivers/librte_mempool_dpaa.a drivers/librte_mempool_dpaa2.a drivers/librte_mempool_octeontx.a drivers/librte_mempool_octeontx2.a drivers/librte_mempool_ring.a drivers/librte_mempool_stack.a drivers/librte_pmd_af_packet.a drivers/librte_pmd_ark.a drivers/librte_pmd_atlantic.a drivers/librte_pmd_avp.a drivers/librte_pmd_axgbe.a drivers/librte_pmd_bond.a drivers/librte_pmd_bnx2x.a drivers/librte_pmd_bnxt.a drivers/librte_pmd_cxgbe.a drivers/librte_pmd_dpaa.a drivers/librte_pmd_dpaa2.a drivers/librte_pmd_e1000.a drivers/librte_pmd_ena.a drivers/librte_pmd_enetc.a drivers/librte_pmd_enic.a drivers/librte_pmd_failsafe.a drivers/librte_pmd_fm10k.a drivers/librte_pmd_i40e.a drivers/librte_pmd_hinic.a drivers/librte_pmd_hns3.a drivers/librte_pmd_iavf.a drivers/librte_pmd_ice.a drivers/librte_pmd_ipn3ke.a drivers/librte_pmd_ixgbe.a drivers/librte_pmd_kni.a drivers/librte_pmd_liquidio.a drivers/librte_pmd_memif.a drivers/librte_pmd_mlx4.a drivers/librte_pmd_mlx5.a drivers/librte_pmd_netvsc.a drivers/librte_pmd_nfp.a drivers/librte_pmd_null.a drivers/librte_pmd_octeontx.a drivers/librte_pmd_octeontx2.a drivers/librte_pmd_pcap.a drivers/librte_pmd_pfe.a drivers/librte_pmd_qede.a drivers/librte_pmd_ring.a drivers/librte_pmd_softnic.a drivers/librte_pmd_tap.a drivers/librte_pmd_thunderx.a drivers/librte_pmd_vdev_netvsc.a drivers/librte_pmd_vhost.a drivers/librte_pmd_virtio.a drivers/librte_pmd_vmxnet3.a drivers/librte_rawdev_dpaa2_cmdif.a drivers/librte_rawdev_dpaa2_qdma.a drivers/librte_rawdev_ifpga.a drivers/librte_rawdev_ntb.a drivers/librte_rawdev_octeontx2_dma.a drivers/librte_rawdev_octeontx2_ep.a drivers/librte_rawdev_skeleton.a drivers/librte_pmd_caam_jr.a drivers/librte_pmd_ccp.a drivers/librte_pmd_dpaa_sec.a drivers/librte_pmd_dpaa2_sec.a drivers/librte_pmd_nitrox.a drivers/librte_pmd_null_crypto.a drivers/librte_pmd_octeontx_crypto.a drivers/librte_pmd_octeontx2_crypto.a drivers/librte_pmd_openssl.a drivers/librte_pmd_crypto_scheduler.a drivers/librte_pmd_virtio_crypto.a drivers/librte_pmd_octeontx_compress.a drivers/librte_pmd_qat.a drivers/librte_pmd_zlib.a drivers/librte_pmd_ifc.a drivers/librte_pmd_mlx5_vdpa.a drivers/librte_pmd_dpaa_event.a drivers/librte_pmd_dpaa2_event.a drivers/librte_pmd_octeontx2_event.a drivers/librte_pmd_opdl_event.a drivers/librte_pmd_skeleton_event.a drivers/librte_pmd_sw_event.a drivers/librte_pmd_dsw_event.a drivers/librte_pmd_octeontx_event.a drivers/librte_pmd_bbdev_null.a drivers/librte_pmd_bbdev_turbo_sw.a drivers/librte_pmd_bbdev_fpga_lte_fec.a -Wl,–no-whole-archive -Wl,–no-as-needed -pthread -lm -ldl -lnuma lib/librte_eal.a lib/librte_kvargs.a lib/librte_mempool.a lib/librte_ring.a lib/librte_net.a lib/librte_mbuf.a lib/librte_ethdev.a lib/librte_meter.a lib/librte_cmdline.a lib/librte_acl.a lib/librte_lpm.a lib/librte_hash.a lib/librte_pci.a lib/librte_cryptodev.a lib/librte_eventdev.a lib/librte_timer.a lib/librte_rawdev.a lib/librte_stack.a lib/librte_sched.a lib/librte_ip_frag.a lib/librte_security.a lib/librte_kni.a lib/librte_pipeline.a lib/librte_port.a lib/librte_table.a lib/librte_gso.a lib/librte_vhost.a lib/librte_reorder.a lib/librte_compressdev.a lib/librte_bbdev.a -lpcap /usr/lib/aarch64-linux-gnu/libbsd.so -lmlx5 -libverbs -Wl,–end-group /usr/lib/aarch64-linux-gnu/libz.so -lmlx4 -libverbs -lmlx5 -libverbs -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap /usr/lib/aarch64-linux-gnu/libfdt.so /usr/lib/aarch64-linux-gnu/libcrypto.so -lmlx5 -libverbs
Disk quota exceeded
I suppose something has been done on Travis side.
1 Like
This is still occurring:
https://travis-ci.com/github/ovsrobot/dpdk/jobs/302408977
Is there something we can do about this?
Same problem for me. I wanted to give ARM64 a try and build fails with the “disk quota” error immediately:
https://travis-ci.com/github/VDR4Arch/vdr4arch/builds/155097951
Is “ARM64” still a thing being worked on?
I am also seeing disk quota exceeded but on ppc64le. I have added df commands these but shows plenty of space.
Here are some examples:https://travis-ci.org/github/djlwilder/ovs/jobs/669791873 https://travis-ci.org/github/djlwilder/ovs/jobs/668945838 https://travis-ci.org/github/djlwilder/dpdk/builds/669007111
These builds run ARM64, X86 and ppc64le jobs, I am only seeing the problem on ppc64le builds.
Thanks
We (dpdk project) will stop using Travis with arches != x86 if there is no visible progress on this.
I have encountered this issue many times during on Arm64 building, but this issue has not occurred before (perhaps a month or two ago)
Another similar reported issue:
We are seeing disk quota exceeded errors on ppc64 builds for the past few days.
Cannot mkdir: Disk quota exceeded
The build has been restarted multiple times, but is failing with the same error.
The failed build log is available here
1 Like
Lots of similar reports on this (I tried to collect links to all here: S390x Adding APT Sources - No usable temporary directory found - /tmp ENOSPC ). I’ve also in the past email Travis support, but no reply, just as there is no reply on this discussion board. They are playing possum… I’d be happy if they at least acknowledged the issue. In the meantime they post a blog post that touts how great their Arm64 support is (see https://blog.travis-ci.com/arm-architecture-adoption-grows-on-travis-ci ), when in reality it is full of bugs and apparently unmaintained.
Damian
May 18, 2020, 2:18pm
10
Sorry for late replay.
3 Likes
Great! How about fixing the other remaining bugs on non-intel platforms, eg the permission issues for python?