Disk quota exceeded on arm64

We have noticed disk quota failures with jobs for the opensource dpdk project since Friday 03/13.

Trying to add some debug like df -h in jobs of mine, I get quite random outputs depending on the environment the job is executed on:

hostname: edb47637-c344-4a03-8538-711046469ad4@14975.lxd-arm64-01-com
Runtime kernel version: 5.3.0-40-generic
/var/snap/lxd/common/lxd/storage-pools/instances/containers/travis-job-david-marchand-dpdk-298848141/rootfs 9.4G 2.7G 6.7G 29% /

hostname: 5f37dcf5-9079-405b-b474-693481472995@25560.lxd-arm64-03-com
Runtime kernel version: 5.3.0-24-generic
/var/snap/lxd/common/lxd/storage-pools/instances/containers/travis-job-david-marchand-dpdk-298855494/rootfs 895G 33G 860G 4% /

hostname: 306788d5-7614-4e9d-856a-1043cb4aeaf4@6138.lxd-arm64-02-com
Runtime kernel version: 5.3.0-40-generic
/var/snap/lxd/common/lxd/storage-pools/instances/containers/travis-job-david-marchand-dpdk-298858463/rootfs 895G 28G 864G 4% /

Is there something changing on arm containers side at the moment?

Is it normal to see those (hw an sw) differences between systems?

1 Like

Latest failure on master branch:
https://travis-ci.com/github/DPDK/dpdk/jobs/299673401#L2951

gcc -o examples/dpdk-vmdq_dcb ‘examples/examples@@dpdk-vmdq_dcb@exe/vmdq_dcb_main.c.o’ -Wl,–no-undefined -Wl,–as-needed -Wl,–whole-archive -Wl,–start-group drivers/librte_common_cpt.a drivers/librte_common_dpaax.a drivers/librte_common_iavf.a drivers/librte_common_mlx5.a drivers/librte_common_octeontx.a drivers/librte_common_octeontx2.a drivers/librte_bus_dpaa.a drivers/librte_bus_fslmc.a drivers/librte_bus_ifpga.a drivers/librte_bus_pci.a drivers/librte_bus_vdev.a drivers/librte_bus_vmbus.a drivers/librte_mempool_bucket.a drivers/librte_mempool_dpaa.a drivers/librte_mempool_dpaa2.a drivers/librte_mempool_octeontx.a drivers/librte_mempool_octeontx2.a drivers/librte_mempool_ring.a drivers/librte_mempool_stack.a drivers/librte_pmd_af_packet.a drivers/librte_pmd_ark.a drivers/librte_pmd_atlantic.a drivers/librte_pmd_avp.a drivers/librte_pmd_axgbe.a drivers/librte_pmd_bond.a drivers/librte_pmd_bnx2x.a drivers/librte_pmd_bnxt.a drivers/librte_pmd_cxgbe.a drivers/librte_pmd_dpaa.a drivers/librte_pmd_dpaa2.a drivers/librte_pmd_e1000.a drivers/librte_pmd_ena.a drivers/librte_pmd_enetc.a drivers/librte_pmd_enic.a drivers/librte_pmd_failsafe.a drivers/librte_pmd_fm10k.a drivers/librte_pmd_i40e.a drivers/librte_pmd_hinic.a drivers/librte_pmd_hns3.a drivers/librte_pmd_iavf.a drivers/librte_pmd_ice.a drivers/librte_pmd_ipn3ke.a drivers/librte_pmd_ixgbe.a drivers/librte_pmd_kni.a drivers/librte_pmd_liquidio.a drivers/librte_pmd_memif.a drivers/librte_pmd_mlx4.a drivers/librte_pmd_mlx5.a drivers/librte_pmd_netvsc.a drivers/librte_pmd_nfp.a drivers/librte_pmd_null.a drivers/librte_pmd_octeontx.a drivers/librte_pmd_octeontx2.a drivers/librte_pmd_pcap.a drivers/librte_pmd_pfe.a drivers/librte_pmd_qede.a drivers/librte_pmd_ring.a drivers/librte_pmd_softnic.a drivers/librte_pmd_tap.a drivers/librte_pmd_thunderx.a drivers/librte_pmd_vdev_netvsc.a drivers/librte_pmd_vhost.a drivers/librte_pmd_virtio.a drivers/librte_pmd_vmxnet3.a drivers/librte_rawdev_dpaa2_cmdif.a drivers/librte_rawdev_dpaa2_qdma.a drivers/librte_rawdev_ifpga.a drivers/librte_rawdev_ntb.a drivers/librte_rawdev_octeontx2_dma.a drivers/librte_rawdev_octeontx2_ep.a drivers/librte_rawdev_skeleton.a drivers/librte_pmd_caam_jr.a drivers/librte_pmd_ccp.a drivers/librte_pmd_dpaa_sec.a drivers/librte_pmd_dpaa2_sec.a drivers/librte_pmd_nitrox.a drivers/librte_pmd_null_crypto.a drivers/librte_pmd_octeontx_crypto.a drivers/librte_pmd_octeontx2_crypto.a drivers/librte_pmd_openssl.a drivers/librte_pmd_crypto_scheduler.a drivers/librte_pmd_virtio_crypto.a drivers/librte_pmd_octeontx_compress.a drivers/librte_pmd_qat.a drivers/librte_pmd_zlib.a drivers/librte_pmd_ifc.a drivers/librte_pmd_mlx5_vdpa.a drivers/librte_pmd_dpaa_event.a drivers/librte_pmd_dpaa2_event.a drivers/librte_pmd_octeontx2_event.a drivers/librte_pmd_opdl_event.a drivers/librte_pmd_skeleton_event.a drivers/librte_pmd_sw_event.a drivers/librte_pmd_dsw_event.a drivers/librte_pmd_octeontx_event.a drivers/librte_pmd_bbdev_null.a drivers/librte_pmd_bbdev_turbo_sw.a drivers/librte_pmd_bbdev_fpga_lte_fec.a -Wl,–no-whole-archive -Wl,–no-as-needed -pthread -lm -ldl -lnuma lib/librte_eal.a lib/librte_kvargs.a lib/librte_mempool.a lib/librte_ring.a lib/librte_net.a lib/librte_mbuf.a lib/librte_ethdev.a lib/librte_meter.a lib/librte_cmdline.a lib/librte_pci.a lib/librte_cryptodev.a lib/librte_hash.a lib/librte_eventdev.a lib/librte_timer.a lib/librte_rawdev.a lib/librte_stack.a lib/librte_sched.a lib/librte_ip_frag.a lib/librte_security.a lib/librte_kni.a lib/librte_pipeline.a lib/librte_port.a lib/librte_table.a lib/librte_lpm.a lib/librte_acl.a lib/librte_gso.a lib/librte_vhost.a lib/librte_reorder.a lib/librte_compressdev.a lib/librte_bbdev.a -lpcap /usr/lib/aarch64-linux-gnu/libbsd.so -lmlx5 -libverbs -Wl,–end-group /usr/lib/aarch64-linux-gnu/libz.so -lmlx4 -libverbs -lmlx5 -libverbs -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap /usr/lib/aarch64-linux-gnu/libfdt.so /usr/lib/aarch64-linux-gnu/libcrypto.so -lmlx5 -libverbs

collect2: error: ld returned 1 exit status

[snip]

FAILED: examples/dpdk-l3fwd-acl

gcc -o examples/dpdk-l3fwd-acl ‘examples/examples@@dpdk-l3fwd-acl@exe/l3fwd-acl_main.c.o’ -Wl,–no-undefined -Wl,–as-needed -Wl,–whole-archive -Wl,–start-group drivers/librte_common_cpt.a drivers/librte_common_dpaax.a drivers/librte_common_iavf.a drivers/librte_common_mlx5.a drivers/librte_common_octeontx.a drivers/librte_common_octeontx2.a drivers/librte_bus_dpaa.a drivers/librte_bus_fslmc.a drivers/librte_bus_ifpga.a drivers/librte_bus_pci.a drivers/librte_bus_vdev.a drivers/librte_bus_vmbus.a drivers/librte_mempool_bucket.a drivers/librte_mempool_dpaa.a drivers/librte_mempool_dpaa2.a drivers/librte_mempool_octeontx.a drivers/librte_mempool_octeontx2.a drivers/librte_mempool_ring.a drivers/librte_mempool_stack.a drivers/librte_pmd_af_packet.a drivers/librte_pmd_ark.a drivers/librte_pmd_atlantic.a drivers/librte_pmd_avp.a drivers/librte_pmd_axgbe.a drivers/librte_pmd_bond.a drivers/librte_pmd_bnx2x.a drivers/librte_pmd_bnxt.a drivers/librte_pmd_cxgbe.a drivers/librte_pmd_dpaa.a drivers/librte_pmd_dpaa2.a drivers/librte_pmd_e1000.a drivers/librte_pmd_ena.a drivers/librte_pmd_enetc.a drivers/librte_pmd_enic.a drivers/librte_pmd_failsafe.a drivers/librte_pmd_fm10k.a drivers/librte_pmd_i40e.a drivers/librte_pmd_hinic.a drivers/librte_pmd_hns3.a drivers/librte_pmd_iavf.a drivers/librte_pmd_ice.a drivers/librte_pmd_ipn3ke.a drivers/librte_pmd_ixgbe.a drivers/librte_pmd_kni.a drivers/librte_pmd_liquidio.a drivers/librte_pmd_memif.a drivers/librte_pmd_mlx4.a drivers/librte_pmd_mlx5.a drivers/librte_pmd_netvsc.a drivers/librte_pmd_nfp.a drivers/librte_pmd_null.a drivers/librte_pmd_octeontx.a drivers/librte_pmd_octeontx2.a drivers/librte_pmd_pcap.a drivers/librte_pmd_pfe.a drivers/librte_pmd_qede.a drivers/librte_pmd_ring.a drivers/librte_pmd_softnic.a drivers/librte_pmd_tap.a drivers/librte_pmd_thunderx.a drivers/librte_pmd_vdev_netvsc.a drivers/librte_pmd_vhost.a drivers/librte_pmd_virtio.a drivers/librte_pmd_vmxnet3.a drivers/librte_rawdev_dpaa2_cmdif.a drivers/librte_rawdev_dpaa2_qdma.a drivers/librte_rawdev_ifpga.a drivers/librte_rawdev_ntb.a drivers/librte_rawdev_octeontx2_dma.a drivers/librte_rawdev_octeontx2_ep.a drivers/librte_rawdev_skeleton.a drivers/librte_pmd_caam_jr.a drivers/librte_pmd_ccp.a drivers/librte_pmd_dpaa_sec.a drivers/librte_pmd_dpaa2_sec.a drivers/librte_pmd_nitrox.a drivers/librte_pmd_null_crypto.a drivers/librte_pmd_octeontx_crypto.a drivers/librte_pmd_octeontx2_crypto.a drivers/librte_pmd_openssl.a drivers/librte_pmd_crypto_scheduler.a drivers/librte_pmd_virtio_crypto.a drivers/librte_pmd_octeontx_compress.a drivers/librte_pmd_qat.a drivers/librte_pmd_zlib.a drivers/librte_pmd_ifc.a drivers/librte_pmd_mlx5_vdpa.a drivers/librte_pmd_dpaa_event.a drivers/librte_pmd_dpaa2_event.a drivers/librte_pmd_octeontx2_event.a drivers/librte_pmd_opdl_event.a drivers/librte_pmd_skeleton_event.a drivers/librte_pmd_sw_event.a drivers/librte_pmd_dsw_event.a drivers/librte_pmd_octeontx_event.a drivers/librte_pmd_bbdev_null.a drivers/librte_pmd_bbdev_turbo_sw.a drivers/librte_pmd_bbdev_fpga_lte_fec.a -Wl,–no-whole-archive -Wl,–no-as-needed -pthread -lm -ldl -lnuma lib/librte_eal.a lib/librte_kvargs.a lib/librte_mempool.a lib/librte_ring.a lib/librte_net.a lib/librte_mbuf.a lib/librte_ethdev.a lib/librte_meter.a lib/librte_cmdline.a lib/librte_acl.a lib/librte_lpm.a lib/librte_hash.a lib/librte_pci.a lib/librte_cryptodev.a lib/librte_eventdev.a lib/librte_timer.a lib/librte_rawdev.a lib/librte_stack.a lib/librte_sched.a lib/librte_ip_frag.a lib/librte_security.a lib/librte_kni.a lib/librte_pipeline.a lib/librte_port.a lib/librte_table.a lib/librte_gso.a lib/librte_vhost.a lib/librte_reorder.a lib/librte_compressdev.a lib/librte_bbdev.a -lpcap /usr/lib/aarch64-linux-gnu/libbsd.so -lmlx5 -libverbs -Wl,–end-group /usr/lib/aarch64-linux-gnu/libz.so -lmlx4 -libverbs -lmlx5 -libverbs -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap -lpcap /usr/lib/aarch64-linux-gnu/libfdt.so /usr/lib/aarch64-linux-gnu/libcrypto.so -lmlx5 -libverbs

Disk quota exceeded

I suppose something has been done on Travis side.
This issue has not been reproduced since 03/20.

1 Like

This is still occurring:

https://travis-ci.com/github/ovsrobot/dpdk/jobs/302408977

Is there something we can do about this?

Same problem for me. I wanted to give ARM64 a try and build fails with the “disk quota” error immediately:

https://travis-ci.com/github/VDR4Arch/vdr4arch/builds/155097951

Is “ARM64” still a thing being worked on?

I am also seeing disk quota exceeded but on ppc64le. I have added df commands these but shows plenty of space.

Here are some examples:
https://travis-ci.org/github/djlwilder/ovs/jobs/669791873
https://travis-ci.org/github/djlwilder/ovs/jobs/668945838
https://travis-ci.org/github/djlwilder/dpdk/builds/669007111

These builds run ARM64, X86 and ppc64le jobs, I am only seeing the problem on ppc64le builds.

Thanks

Imprint