cmis-in-batch released with data generation mode

I have just pushed proper 1.0 release of the cmis-in-batch tool to github and bintray:

https://github.com/karolbe/cmis-in-batch

https://bintray.com/karolbe/metasys/cmis-in-batch

Here is a quote from the README.md:

Data generation is a useful feature that allows bulk importing of test data documents into CMIS compatible repository. Additionally it can populate metadata of documents with values coming from predefined dictionaries.

Sample script for generating thousands of documents can look like this one below.

Here is a brief description of what the script does:

* it will load three dictionaries from files /tmp/disciplines, /tmp/types, /tmp/subtypes. The dictionaries are simple text files where values are separated by new line characters. From the dictionary values Cartesian product will be calculated so for example, having three dictionaries:

1. level1A, level1B
2. level2A, level2B
3. level3A, level3B
following combinations will be generated:


[level1A, level2A, level3A]
[level1A, level2A, level3B]
[level1A, level2B, level3A]
[level1A, level2B, level3B]
[level1B, level2A, level3A]
[level1B, level2A, level3B]
[level1B, level2B, level3A]
[level1B, level2B, level3B]

* it will import each file in content-path location (“/media/kbryd/Media/work/sample_data/department”) to a location in repository defined with linking-rule: /Repository/${discipline}/static/${doctype}/sub/${docsubtype} – each ${} variable will be replaced by a value coming from appropriate dictionary.
* naming-rule defines what the object name should be. It can use variables from the dictionaries plus a few additional: ${file_name}, ${file_size}, ${file_path}, ${file_ext}, ${file_mime}
* mapping defines the mapping for populating metadata of each document, e.g. in this case discipline attribute will be populated with value of discipline.


   generate-random-data "set1" {
   doc-type "cara_document"
   linking-rule "/Repository/${discipline}/static/${doctype}/sub/${docsubtype}"
   naming-rule "${file_name} - ${doctype}"
   content-path "/media/kbryd/Media/work/sample_data/department"

   mapping {
       discipline {
          "${discipline}"
       }
       doc_type {
          "${doctype}"
       }
       doc_subtype {
          "${docsubtype}"
       }
   }

   dictionaries {
      discipline "/tmp/disciplines"
      doctype "/tmp/types"
      docsubtype "/tmp/subtypes"
   }
}

And that’s all! Have fun using it! 🙂

Docker & orphaned volumes

What is the best way to get rid of orphaned Docker volumes?

I have written one liner that seems to be solving this problem. First lets check how many volumes there are in total:


root@quad:~# docker volume ls | wc -l
72

72 volumes. Quite a lot to be honest.

Then I run this to double check that it really does return valid data.


for vol in `docker volume ls -qf dangling=true`; do  docker volume inspect $vol | jq -r '.[] | .Mountpoint' | xargs ls ; done

and I get 52 lines like these:


ls: cannot access '/opt/docker/volumes/07a1d0711833af526d08809118ae880fce4d6c537d4246c2fc332fceca0cde9a/_data': No such file or directory
ls: cannot access '/opt/docker/volumes/0d88bab60c8f5023a22f3ff15b35ab585c66b476f2e6615db06f50fefb2bc970/_data': No such file or directory
ls: cannot access '/opt/docker/volumes/0e4e14e289b25a65d0ee597bc7810c6f09961d13a059293a357c425b46534f5a/_data': No such file or directory
ls: cannot access '/opt/docker/volumes/18c2481850150cab1c46b0bdc9178e959c05dd4a205a45510c20467c6b95e473/_data': No such file or directory
ls: cannot access '/opt/docker/volumes/2275d260b670d10d026f1be776a9d32b9a26e23d513cca63add039e9451e3454/_data': No such file or directory
ls: cannot access '/opt/docker/volumes/28252f4885f02615e33d1ed2fdc4b45ebb21d38b933457760090f9f3b68f4e09/_data': No such file or directory

which means that only 20 volumes are actually valid, rest of them point to nonexistent folders on the hard drive.

So, they can be easily removed by running


docker volume ls -qf dangling=true | | xargs -r docker volume rm