luigi.contrib.hdfs package¶
Submodules¶
- luigi.contrib.hdfs.abstract_client module
- luigi.contrib.hdfs.clients module
- luigi.contrib.hdfs.config module
- luigi.contrib.hdfs.error module
- luigi.contrib.hdfs.format module
- luigi.contrib.hdfs.hadoopcli_clients module
create_hadoopcli_client()
HdfsClient
HdfsClient.recursive_listdir_cmd
HdfsClient.call_check()
HdfsClient.exists()
HdfsClient.move()
HdfsClient.remove()
HdfsClient.chmod()
HdfsClient.chown()
HdfsClient.count()
HdfsClient.copy()
HdfsClient.put()
HdfsClient.get()
HdfsClient.getmerge()
HdfsClient.mkdir()
HdfsClient.listdir()
HdfsClient.touchz()
HdfsClientCdh3
HdfsClientApache1
- luigi.contrib.hdfs.target module
- luigi.contrib.hdfs.webhdfs_client module
webhdfs
WebHdfsClient
WebHdfsClient.url
WebHdfsClient.client
WebHdfsClient.walk()
WebHdfsClient.exists()
WebHdfsClient.upload()
WebHdfsClient.download()
WebHdfsClient.remove()
WebHdfsClient.read()
WebHdfsClient.move()
WebHdfsClient.mkdir()
WebHdfsClient.chmod()
WebHdfsClient.chown()
WebHdfsClient.count()
WebHdfsClient.copy()
WebHdfsClient.put()
WebHdfsClient.get()
WebHdfsClient.listdir()
WebHdfsClient.touchz()
Module contents¶
Provides access to HDFS using the HdfsTarget
, a subclass of Target
.
You can configure what client by setting the “client” config under the “hdfs” section in the configuration, or using the --hdfs-client
command line option.
“hadoopcli” is the slowest, but should work out of the box.
Since the hdfs functionality is quite big in luigi, it’s split into smaller
files under luigi/contrib/hdfs/*.py
. But for the sake of convenience and
API stability, everything is reexported under luigi.contrib.hdfs
.