You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "liangsibin (JIRA)" <ji...@apache.org> on 2017/02/25 13:22:44 UTC

[jira] [Created] (CASSANDRA-13266) Bulk loading sometimes is very slow?

liangsibin created CASSANDRA-13266:
--------------------------------------

             Summary: Bulk loading sometimes is very slow?
                 Key: CASSANDRA-13266
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13266
             Project: Cassandra
          Issue Type: Improvement
            Reporter: liangsibin


When I bulkload sstable created with CQLSSTableWriter, it's sometimes very slow.  
use 2 nodes write SSTable and bulkload
1、Use CQLSSTableWriter create SSTable (60 threads)
2、When the directory over 100000 rows,bulkload the directory (20 threads)
the normal bulkload speed is about 70M/s per node,and bulkload 141G SStables cost 90 minutes but sometimes is very slow,the same data cost 4 hours why?
here is the code bulkload sstable

public class JmxBulkLoader {
	
    static final Logger LOGGER = LoggerFactory.getLogger(JmxBulkLoader.class);
	private JMXConnector connector;
	private StorageServiceMBean storageBean;
	private Timer timer = new Timer();

	public JmxBulkLoader(String host, int port) throws Exception {
		connect(host, port);
	}


	private void connect(String host, int port) throws IOException, MalformedObjectNameException {
		JMXServiceURL jmxUrl = new JMXServiceURL(
				String.format("service:jmx:rmi:///jndi/rmi://%s:%d/jmxrmi", host, port));
		Map<String, Object> env = new HashMap<String, Object>();
		connector = JMXConnectorFactory.connect(jmxUrl, env);
		MBeanServerConnection mbeanServerConn = connector.getMBeanServerConnection();
		ObjectName name = new ObjectName("org.apache.cassandra.db:type=StorageService");
		storageBean = JMX.newMBeanProxy(mbeanServerConn, name, StorageServiceMBean.class);
	}

	public void close() throws IOException {
		connector.close();
	}

	public void bulkLoad(String path) {
		LOGGER.info("begin load data to cassandra " + new Path(path).getName());
		timer.start();
		storageBean.bulkLoad(path);
		timer.end();
		LOGGER.info("bulk load took " + timer.getTimeTakenMillis() + "ms, path: " + new Path(path).getName());
	}
}

bulkload thread 
public class BulkThread implements Runnable {

	private String path;
	private String jmxHost;
	private int jmxPort;
	
	public BulkThread(String path, String jmxHost, int jmxPort) {
		super();
		this.path = path;
		this.jmxHost = jmxHost;
		this.jmxPort = jmxPort;
	}
	@Override
	public void run() {
		JmxBulkLoader bulkLoader = null;
		try {
			bulkLoader = new JmxBulkLoader(jmxHost, jmxPort);
			bulkLoader.bulkLoad(path);
		} catch (Exception e) {
			e.printStackTrace();
		} finally {
			if (bulkLoader != null)
				try {
					bulkLoader.close();
					bulkLoader = null;
				} catch (IOException e) {
					e.printStackTrace();
				}
		}
	}
}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)