Disk quota exceeded on arm64

We have noticed disk quota failures with jobs for the opensource dpdk project since Friday 03/13.

Trying to add some debug like df -h in jobs of mine, I get quite random outputs depending on the environment the job is executed on:

hostname: edb47637-c344-4a03-8538-711046469ad4@14975.lxd-arm64-01-com
Runtime kernel version: 5.3.0-40-generic
/var/snap/lxd/common/lxd/storage-pools/instances/containers/travis-job-david-marchand-dpdk-298848141/rootfs 9.4G 2.7G 6.7G 29% /

hostname: 5f37dcf5-9079-405b-b474-693481472995@25560.lxd-arm64-03-com
Runtime kernel version: 5.3.0-24-generic
/var/snap/lxd/common/lxd/storage-pools/instances/containers/travis-job-david-marchand-dpdk-298855494/rootfs 895G 33G 860G 4% /

hostname: 306788d5-7614-4e9d-856a-1043cb4aeaf4@6138.lxd-arm64-02-com
Runtime kernel version: 5.3.0-40-generic
/var/snap/lxd/common/lxd/storage-pools/instances/containers/travis-job-david-marchand-dpdk-298858463/rootfs 895G 28G 864G 4% /

Is there something changing on arm containers side at the moment?

Is it normal to see those (hw an sw) differences between systems?

1 Like

Latest failure on master branch:
https://travis-ci.com/github/DPDK/dpdk/jobs/299673401#L2951

gcc -o examples/dpdk-vmdq_dcb ‘examples/examples@@dpdk-vmdq_dcb@exe/vmdq_dcb_main.c.o’ -Wl,–no-undefined -Wl,–as-needed -Wl,–whole-archive -Wl,–start-group drivers/librte_common_cpt.a drivers/librte_common_dpaax.a drivers/librte_common_iavf.a drivers/librte_common_mlx5.a drivers/librte_common_octeontx.a drivers/librte_common_octeontx2.a drivers/librte_bus_dpaa.a drivers/librte_bus_fslmc.a drivers/librte_bus_ifpga.a drivers/librte_bus_pci.a drivers/librte_bus_vdev.a drivers/librte_bus_vmbus.a drivers/librte_mempool_bucket.a drivers/librte_mempool_dpaa.a drivers/librte_mempool_dpaa2.a drivers/librte_mempool_octeontx.a drivers/librte_mempool_octeontx2.a drivers/librte_mempool_ring.a drivers/librte_mempool_stack.a drivers/librte_pmd_af_packet.a drivers/librte_pmd_ark.a drivers/librte_pmd_atlantic.a drivers/librte_pmd_avp.a drivers/librte_pmd_axgbe.a drivers/librte_pmd_bond.a drivers/librte_pmd_bnx2x.a drivers/librte_pmd_bnxt.a drivers/librte_pmd_cxgbe.a drivers/librte_pmd_dpaa.a drivers/librte_pmd_dpaa2.a drivers/librte_pmd_e1000.a drivers/librte_pmd_ena.a drivers/librte_pmd_enetc.a drivers/librte_pmd_enic.a drivers/librte_pmd_failsafe.a drivers/librte_pmd_fm10k.a drivers/librte_pmd_i40e.a drivers/librte_pmd_hinic.a drivers/librte_pmd_hns3.a drivers/librte_pmd_iavf.a drivers/librte_pmd_ice.a drivers/librte_pmd_ipn3ke.a drivers/librte_pmd_ixgbe.a drivers/librte_pmd_kni.a drivers/librte_pmd_liquidio.a drivers/librte_pmd_memif.a drivers/librte_pmd_mlx4.a drivers/librte_pmd_mlx5.a drivers/librte_pmd_netvsc.a drivers/librte_pmd_nfp.a drivers/librte_pmd_null.a drivers/librte_pmd_octeontx.a drivers/librte_pmd_octeontx2.a drivers/librte_pmd_pcap.a drivers/librte_pmd_pfe.a drivers/librte_pmd_qede.a drivers/librte_pmd_ring.a drivers/librte_pmd_softnic.a drivers/librte_pmd_tap.a drivers/librte_pmd_thunderx.a drivers/librte_pmd_vdev_netvsc.a drivers/librte_pmd_vhost.a drivers/librte_pmd_virtio.a drivers/librte_pmd_vmxnet3.a drivers/librte_rawdev_dpaa2_cmdif.a drivers/librte_rawdev_dpaa2_qdma.a drivers/librte_rawdev_ifpga.a drivers/librte_rawdev_ntb.a drivers/librte_rawdev_octeontx2_dma.a drivers/librte_rawdev_octeontx2_ep.a drivers/librte_rawdev_skeleton.a drivers/librte_pmd_caam_jr.a drivers/librte_pmd_ccp.a drivers/librte_pmd_dpaa_sec.a drivers/librte_pmd_dpaa2_sec.a drivers/librte_pmd_nitrox.a drivers/librte_pmd_null_crypto.a drivers/librte_pmd_octeontx_crypto.a drivers/librte_pmd_octeontx2_crypto.a drivers/librte_pmd_openssl.a drivers/librte_pmd_crypto_scheduler.a drivers/librte_pmd_virtio_crypto.a drivers/librte_pmd_octeontx_compress.a drivers/librte_pmd_qat.a drivers/librte_pmd_zlib.a drivers/librte_pmd_ifc.a drivers/librte_pmd_mlx5_vdpa.a drivers/librte_pmd_dpaa_event.a drivers/librte_pmd_dpaa2_event.a drivers/librte_pmd_octeontx2_event.a drivers/librte_pmd_opdl_event.a drivers/librte_pmd_skeleton_event.a drivers/librte_pmd_sw_event.a drivers/librte_pmd_dsw_event.a drivers/librte_pmd_octeontx_event.a drivers/librte_pmd_bbdev_null.a drivers/librte_pmd_bbdev_turbo_sw.a drivers/librte_pmd_bbdev_fpga_lte_fec.a -Wl,–no-whole-archive -Wl,–no-as-needed -pthread -lm -ldl -lnuma lib/librte_eal.a lib/librte_kvargs.a lib/librte_mempool.a lib/librte_ring.a lib/librte_net.a lib/librte_mbuf.a lib/librte_ethdev.a lib/librte_meter.a lib/librte_cmdline.a lib/librte_pci.a lib/librte_cryptodev.a lib/librte_hash.a lib/librte_eventdev.a lib/librte_timer.a lib/librte_rawdev.a lib/librte_stack.a lib/librte_sched.a lib/librte_ip_frag.a lib/librte_security.a lib/librte_kni.a lib/librte_pipeline.a lib/librte_port.a lib/librte_table.a lib/librte_lpm.a lib/librte_acl.a lib/librte_gso.a lib/librte_vhost.a lib/librte_reorder.a lib/librte_compressdev.a lib/librte_bbdev.a -lpcap /usr/lib/aarch64-linux-gnu/libbsd.so -lmlx5 -libverbs -Wl,–end-group /usr/lib/aarch64-linux-gnu/libz.so -lmlx4 -libverbs -lmlx5 -libverbs -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap /usr/lib/aarch64-linux-gnu/libfdt.so /usr/lib/aarch64-linux-gnu/libcrypto.so -lmlx5 -libverbs

collect2: error: ld returned 1 exit status

[snip]

FAILED: examples/dpdk-l3fwd-acl

gcc -o examples/dpdk-l3fwd-acl ‘examples/examples@@dpdk-l3fwd-acl@exe/l3fwd-acl_main.c.o’ -Wl,–no-undefined -Wl,–as-needed -Wl,–whole-archive -Wl,–start-group drivers/librte_common_cpt.a drivers/librte_common_dpaax.a drivers/librte_common_iavf.a drivers/librte_common_mlx5.a drivers/librte_common_octeontx.a drivers/librte_common_octeontx2.a drivers/librte_bus_dpaa.a drivers/librte_bus_fslmc.a drivers/librte_bus_ifpga.a drivers/librte_bus_pci.a drivers/librte_bus_vdev.a drivers/librte_bus_vmbus.a drivers/librte_mempool_bucket.a drivers/librte_mempool_dpaa.a drivers/librte_mempool_dpaa2.a drivers/librte_mempool_octeontx.a drivers/librte_mempool_octeontx2.a drivers/librte_mempool_ring.a drivers/librte_mempool_stack.a drivers/librte_pmd_af_packet.a drivers/librte_pmd_ark.a drivers/librte_pmd_atlantic.a drivers/librte_pmd_avp.a drivers/librte_pmd_axgbe.a drivers/librte_pmd_bond.a drivers/librte_pmd_bnx2x.a drivers/librte_pmd_bnxt.a drivers/librte_pmd_cxgbe.a drivers/librte_pmd_dpaa.a drivers/librte_pmd_dpaa2.a drivers/librte_pmd_e1000.a drivers/librte_pmd_ena.a drivers/librte_pmd_enetc.a drivers/librte_pmd_enic.a drivers/librte_pmd_failsafe.a drivers/librte_pmd_fm10k.a drivers/librte_pmd_i40e.a drivers/librte_pmd_hinic.a drivers/librte_pmd_hns3.a drivers/librte_pmd_iavf.a drivers/librte_pmd_ice.a drivers/librte_pmd_ipn3ke.a drivers/librte_pmd_ixgbe.a drivers/librte_pmd_kni.a drivers/librte_pmd_liquidio.a drivers/librte_pmd_memif.a drivers/librte_pmd_mlx4.a drivers/librte_pmd_mlx5.a drivers/librte_pmd_netvsc.a drivers/librte_pmd_nfp.a drivers/librte_pmd_null.a drivers/librte_pmd_octeontx.a drivers/librte_pmd_octeontx2.a drivers/librte_pmd_pcap.a drivers/librte_pmd_pfe.a drivers/librte_pmd_qede.a drivers/librte_pmd_ring.a drivers/librte_pmd_softnic.a drivers/librte_pmd_tap.a drivers/librte_pmd_thunderx.a drivers/librte_pmd_vdev_netvsc.a drivers/librte_pmd_vhost.a drivers/librte_pmd_virtio.a drivers/librte_pmd_vmxnet3.a drivers/librte_rawdev_dpaa2_cmdif.a drivers/librte_rawdev_dpaa2_qdma.a drivers/librte_rawdev_ifpga.a drivers/librte_rawdev_ntb.a drivers/librte_rawdev_octeontx2_dma.a drivers/librte_rawdev_octeontx2_ep.a drivers/librte_rawdev_skeleton.a drivers/librte_pmd_caam_jr.a drivers/librte_pmd_ccp.a drivers/librte_pmd_dpaa_sec.a drivers/librte_pmd_dpaa2_sec.a drivers/librte_pmd_nitrox.a drivers/librte_pmd_null_crypto.a drivers/librte_pmd_octeontx_crypto.a drivers/librte_pmd_octeontx2_crypto.a drivers/librte_pmd_openssl.a drivers/librte_pmd_crypto_scheduler.a drivers/librte_pmd_virtio_crypto.a drivers/librte_pmd_octeontx_compress.a drivers/librte_pmd_qat.a drivers/librte_pmd_zlib.a drivers/librte_pmd_ifc.a drivers/librte_pmd_mlx5_vdpa.a drivers/librte_pmd_dpaa_event.a drivers/librte_pmd_dpaa2_event.a drivers/librte_pmd_octeontx2_event.a drivers/librte_pmd_opdl_event.a drivers/librte_pmd_skeleton_event.a drivers/librte_pmd_sw_event.a drivers/librte_pmd_dsw_event.a drivers/librte_pmd_octeontx_event.a drivers/librte_pmd_bbdev_null.a drivers/librte_pmd_bbdev_turbo_sw.a drivers/librte_pmd_bbdev_fpga_lte_fec.a -Wl,–no-whole-archive -Wl,–no-as-needed -pthread -lm -ldl -lnuma lib/librte_eal.a lib/librte_kvargs.a lib/librte_mempool.a lib/librte_ring.a lib/librte_net.a lib/librte_mbuf.a lib/librte_ethdev.a lib/librte_meter.a lib/librte_cmdline.a lib/librte_acl.a lib/librte_lpm.a lib/librte_hash.a lib/librte_pci.a lib/librte_cryptodev.a lib/librte_eventdev.a lib/librte_timer.a lib/librte_rawdev.a lib/librte_stack.a lib/librte_sched.a lib/librte_ip_frag.a lib/librte_security.a lib/librte_kni.a lib/librte_pipeline.a lib/librte_port.a lib/librte_table.a lib/librte_gso.a lib/librte_vhost.a lib/librte_reorder.a lib/librte_compressdev.a lib/librte_bbdev.a -lpcap /usr/lib/aarch64-linux-gnu/libbsd.so -lmlx5 -libverbs -Wl,–end-group /usr/lib/aarch64-linux-gnu/libz.so -lmlx4 -libverbs -lmlx5 -libverbs -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap /usr/lib/aarch64-linux-gnu/libfdt.so /usr/lib/aarch64-linux-gnu/libcrypto.so -lmlx5 -libverbs

Disk quota exceeded

I suppose something has been done on Travis side.
This issue has not been reproduced since 03/20.

1 Like

This is still occurring:

https://travis-ci.com/github/ovsrobot/dpdk/jobs/302408977

Is there something we can do about this?

Same problem for me. I wanted to give ARM64 a try and build fails with the “disk quota” error immediately:

https://travis-ci.com/github/VDR4Arch/vdr4arch/builds/155097951

Is “ARM64” still a thing being worked on?

I am also seeing disk quota exceeded but on ppc64le. I have added df commands these but shows plenty of space.

Here are some examples:
https://travis-ci.org/github/djlwilder/ovs/jobs/669791873
https://travis-ci.org/github/djlwilder/ovs/jobs/668945838
https://travis-ci.org/github/djlwilder/dpdk/builds/669007111

These builds run ARM64, X86 and ppc64le jobs, I am only seeing the problem on ppc64le builds.

Thanks

We (dpdk project) will stop using Travis with arches != x86 if there is no visible progress on this.
Is there any ETA? feedback on the issue we reported 2 months ago?

I have encountered this issue many times during on Arm64 building, but this issue has not occurred before (perhaps a month or two ago)

https://travis-ci.com/github/iecedge/cilium/jobs/330875210#L1900

Another similar reported issue:

1 Like

Lots of similar reports on this (I tried to collect links to all here: S390x Adding APT Sources - No usable temporary directory found - /tmp ENOSPC). I’ve also in the past email Travis support, but no reply, just as there is no reply on this discussion board. They are playing possum… I’d be happy if they at least acknowledged the issue. In the meantime they post a blog post that touts how great their Arm64 support is (see https://blog.travis-ci.com/arm-architecture-adoption-grows-on-travis-ci), when in reality it is full of bugs and apparently unmaintained.

Sorry for late replay.
We have identified and fixed the root cause of disk quota issue on arm64.
There were temporary problems on all arm64 hosts and later on one only.

3 Likes

Great! How about fixing the other remaining bugs on non-intel platforms, eg the permission issues for python?

Imprint