{"id":888,"date":"2014-06-25T06:10:20","date_gmt":"2014-06-25T06:10:20","guid":{"rendered":"http:\/\/poojanwagh.opalstacked.com\/techblog\/?p=888"},"modified":"2014-06-25T06:28:06","modified_gmt":"2014-06-25T06:28:06","slug":"making-duplicity-and-ncftpput-play-nice-with-vsftpd-ftp-daemon","status":"publish","type":"post","link":"https:\/\/tech.poojanblog.com\/blog\/unix-linux\/making-duplicity-and-ncftpput-play-nice-with-vsftpd-ftp-daemon\/","title":{"rendered":"Making duplicity and ncftpput play nice with vsftpd FTP daemon"},"content":{"rendered":"

I’ve been setting up yet another remote backup lately (see associated problem here<\/a>). For this purpose (on Unix), the duplicity<\/a> solution looks ideal. However, I’ve tried it on a couple of lightweight FTP servers (a TL-WDR3600 and a Raspberry Pi) and neither of them work. I keep getting a “Permission Denied” message.<\/p>\n

I did quite a bit of investigation, and found that it’s related to the way that duplicity’s FTP backend, ncftp\/ncftpput<\/a> sends files. Or really, a quirk of vsftpd, where it won’t let you put a file with an absolute path (and more specifically a file with a directory name). I’ve seen other reports of this on the Internet here<\/a> and here<\/a>.<\/p>\n

Here’s a sample ncftpput debug log:<\/p>\n

2014-06-25 00:56:34\u00a0 331: Please specify the password.
\n2014-06-25 00:56:34\u00a0 Cmd: PASS xxxxxxxx
\n2014-06-25 00:56:34\u00a0 230: Login successful.
\n2014-06-25 00:56:34\u00a0 Cmd: PWD
\n2014-06-25 00:56:34\u00a0 257: \"\/\"
\n2014-06-25 00:56:34\u00a0 Logged in to 192.168.1.2 as Poojan.
\n2014-06-25 00:56:34\u00a0 Cmd: FEAT
\n2014-06-25 00:56:34\u00a0 211: Features:
\n2014-06-25 00:56:34\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 EPRT
\n2014-06-25 00:56:34\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 EPSV
\n2014-06-25 00:56:34\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 MDTM
\n2014-06-25 00:56:34\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 PASV
\n2014-06-25 00:56:34\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 REST STREAM
\n2014-06-25 00:56:34\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 SIZE
\n2014-06-25 00:56:34\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 TVFS
\n2014-06-25 00:56:34\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 UTF8
\n2014-06-25 00:56:34\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 End
\n2014-06-25 00:56:34\u00a0 Cmd: PWD
\n2014-06-25 00:56:34\u00a0 257: \"\/\"
\n2014-06-25 00:56:34\u00a0 Cmd: CWD folder1\/duplicity\/Public
\n2014-06-25 00:56:34\u00a0 250: Directory successfully changed.
\n2014-06-25 00:56:34\u00a0 Cmd: CWD \/
\n2014-06-25 00:56:34\u00a0 250: Directory successfully changed.
\n2014-06-25 00:56:34\u00a0 Cmd: TYPE I
\n2014-06-25 00:56:34\u00a0 200: Switching to Binary mode.
\n2014-06-25 00:56:34\u00a0 Cmd: PASV
\n2014-06-25 00:56:34\u00a0 227: Entering Passive Mode (192,168,1,2,225,89).
\n2014-06-25 00:56:34\u00a0 Cmd: STOR folder1\/duplicity\/Public\/duplicity-full.20140625T055614Z.vol1.difftar.gpg
\n2014-06-25 00:56:34\u00a0 550: Permission denied.
\n2014-06-25 00:56:34\u00a0 ncftpput folder1\/duplicity\/Public\/duplicity-full.20140625T055614Z.vol1.difftar.gpg: server said: Permission denied.
\n2014-06-25 00:56:34\u00a0 Cmd: QUIT
\n2014-06-25 00:56:34\u00a0 221: Goodbye.
\n<\/code><\/p>\n

However, I ran a few tests, and I found that if one cd<\/code>‘es to the directory and then puts (STOR<\/code>‘es) the file, everything works. My guess this is a chroot-style feature. So, I created a small Python script that will parse the arguments that duplicity sends, and inserts cd (to the target directory) command before uploading the file. This works well. Here’s the script:<\/p>\n


\n#!\/usr\/local\/bin\/python3
\nimport argparse
\nimport subprocess
\nimport sys
\nimport os<\/p>\n

# ncftpput looks something like this:
\n# 'ncftpput -f \/tmp\/duplicity-yRftm3-tempdir\/mkstemp-PiYFTL-1 -F -t 30 -o useCLNT=0,useHELP_SITE=0\u00a0 -m -V -C '\/tmp\/duplicity-yRftm3-tempdir\/mktemp-sWc1AA-3' 'folder1\/duplicity\/Public\/duplicity-full.20140625T041147Z.vol1.difftar.gz
\nif __name__ == '__main__':
\n parser = argparse.ArgumentParser(description='wrapper for ncftpput command, to allow absolute target paths on vsftpd')
\n parser.add_argument('-f', dest='credentials_file')
\n parser.add_argument('-F', action='store_true')
\n parser.add_argument('-t', dest='timeout')
\n parser.add_argument('-o', dest='ftp_options')
\n parser.add_argument('-m', action='store_true')
\n parser.add_argument('-C', action='store_true')
\n parser.add_argument('-V', action='store_true')
\n parser.add_argument('source')
\n parser.add_argument('dest')
\n args = parser.parse_args()<\/p>\n

# print(args)<\/p>\n

dest_dir = os.path.dirname(args.dest)
\n dest_file = os.path.basename(args.dest)<\/p>\n

#cmd_args = ['\/usr\/local\/bin\/ncftpput', '-d', 'ftp-debug.log', '-W', 'cd {0}'.format(dest_dir)] +\u00a0 sys.argv[1:-1] + [dest_file]
\n cmd_args = ['\/usr\/local\/bin\/ncftpput', '-W', 'cd {0}'.format(dest_dir)] +\u00a0 sys.argv[1:-1] + [dest_file]
\n subprocess.Popen(cmd_args)
\n<\/code><\/p>\n

The script is called ncftpput<\/code>, and is placed in a directory called ~\/duplicity_bin. I then add this directory to my path before running duplicity. Here’s a shell script that does that:<\/p>\n


\n#!\/bin\/sh
\nexport FTP_PASSWORD=\"XXXYYYXXYXYZY\"
\nexport PATH=\"\/home\/Poojan\/duplicity_bin:$PATH\"
\nduplicity -v 9 --encrypt-key=DEADBEEF full \/tank\/Users\/Public ftp:\/\/Poojan@192.168.1.2\/folder1\/duplicity\/Public
\n<\/code>\n<\/p>\n