You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by "blackerby (via GitHub)" <gi...@apache.org> on 2024/04/11 16:04:08 UTC
[I] Cannot query sitemap.xml files (drill)
blackerby opened a new issue, #2899:
URL: https://github.com/apache/drill/issues/2899
**Describe the bug**
Querying sitemaps file "succeeds" but the web UI says no data is available.
**To Reproduce**
Steps to reproduce the behavior:
1. Create a local copy of https://www.govinfo.gov/sitemap/bulkdata/PLAW/118publ/sitemap.xml on your hard drive.
2. Query the file
```sql
select * from dfs.`<path_to_file>/sitemap.xml`;
```
3. See error in the screenshot below
**Expected behavior**
I expect to see data in the output table.
**Error detail, log output or screenshots**
**Drill version**
1.21.1
(cc: @cgivre)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot query sitemap.xml files (drill)
Posted by "blackerby (via GitHub)" <gi...@apache.org>.
blackerby commented on issue #2899:
URL: https://github.com/apache/drill/issues/2899#issuecomment-2050486963
Similar issue when I try to query this MODS file
https://www.govinfo.gov/metadata/pkg/PLAW-106publ246/mods.xml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot query sitemap.xml files (drill)
Posted by "cgivre (via GitHub)" <gi...@apache.org>.
cgivre commented on issue #2899:
URL: https://github.com/apache/drill/issues/2899#issuecomment-2080144620
@blackerby I found the cause for this issues and submitted a fix. Would you mind please testing it out and submitting a review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot query sitemap.xml files (drill)
Posted by "blackerby (via GitHub)" <gi...@apache.org>.
blackerby commented on issue #2899:
URL: https://github.com/apache/drill/issues/2899#issuecomment-2080760709
@cgivre Sure thing, should be able to do so in the next day or two
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot query sitemap.xml files (drill)
Posted by "cgivre (via GitHub)" <gi...@apache.org>.
cgivre closed issue #2899: Cannot query sitemap.xml files
URL: https://github.com/apache/drill/issues/2899
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot query sitemap.xml files (drill)
Posted by "blackerby (via GitHub)" <gi...@apache.org>.
blackerby commented on issue #2899:
URL: https://github.com/apache/drill/issues/2899#issuecomment-2050568285
Removing all attributes from the root elements in the sitemap and MODS files results in data being returned
Copy/paste isn't really cooperating with my MODS example, but here is the output after removing attributes from the root element.
```bash
apache drill> select * from dfs.`/Users/wm/Desktop/sitemap.xml` t;
+------------+----------------------------------------------------------------------------------+
| attributes | url |
+------------+----------------------------------------------------------------------------------+
| {} | {"loc":"https://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ1.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ2.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ3.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ4.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ5.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ6.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ8.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ7.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ9.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ11.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ12.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ10.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ13.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ14.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/p
ublic/PLAW-118publ16.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ18.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ19.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ17.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ15.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ21.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ20.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ22.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ23.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ30.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ28.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ29.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ26.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ25.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ24.xmlhttps://www.govinfo.gov/bulkdata/PL
AW/118/public/PLAW-118publ27.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ34.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ33.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ32.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ35.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ037.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ36.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ37.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ38.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ39.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ31.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ40.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ41.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ45.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ44.xmlhttps://www.govinfo.gov/bu
lkdata/PLAW/118/public/PLAW-118publ46.xmlhttps://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ43.xml","lastmod":"2024-03-28T00:10:00.074Z2023-06-20T23:44:00.215Z2023-07-03T14:32:01.529Z2023-08-29T19:17:07.501Z2023-12-11T20:04:05.659Z2023-08-29T19:18:04.606Z2023-08-29T19:45:02.701Z2023-12-11T19:11:06.925Z2023-12-11T19:12:10.757Z2024-02-15T16:27:03.296Z2023-12-11T19:47:04.743Z2024-02-15T16:27:03.236Z2024-02-15T16:27:03.265Z2024-02-15T15:44:41.386Z2023-12-11T20:04:05.922Z2023-12-11T20:05:08.683Z2023-12-12T00:49:02.710Z2024-02-15T15:44:41.414Z2024-02-15T15:44:41.443Z2024-02-15T15:44:41.471Z2023-11-21T16:53:01.237Z2023-12-07T23:10:02.343Z2024-01-04T16:42:00.154Z2024-01-04T16:42:00.247Z2024-01-04T16:42:00.275Z2024-01-04T16:42:00.340Z2024-01-04T16:42:00.369Z2024-01-04T16:42:00.397Z2024-01-09T15:48:00.645Z2024-02-14T00:54:00.081Z2024-02-07T19:55:17.124Z2024-01-22T13:57:00.039Z2024-01-22T19:17:00.607Z2024-01-24T22:17:22.372Z2024-02-02T13:21:00.064Z2024-02-02T13:21:00.142Z2024-02-12T2
0:18:53.802Z2024-02-15T13:17:00.671Z2024-02-21T13:38:00.160Z2024-03-14T16:09:00.050Z2024-03-11T12:59:00.649Z2024-03-15T12:16:00.039Z2024-03-29T12:26:00.117Z2024-03-29T12:26:00.173Z2024-03-29T12:26:00.209Z2024-04-01T12:39:00.068Z","changefreq":"monthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthlymonthly","priority":"1.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.01.0"} |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Cannot query sitemap.xml files (drill)
Posted by "blackerby (via GitHub)" <gi...@apache.org>.
blackerby commented on issue #2899:
URL: https://github.com/apache/drill/issues/2899#issuecomment-2080889935
Alright, got your fork up and running on my machine. Borrowing a little syntax from the added test (I'm still learning Drill), this worked perfectly:
```sql
apache drill> select * from table(dfs.`/Users/wm/Desktop/sitemap.xml` (type => 'xml', dataLevel => 2)) limit 5;
+------------+--------------------------------------------------------------------+--------------------------+------------+----------+
| attributes | loc | lastmod | changefreq | priority |
+------------+--------------------------------------------------------------------+--------------------------+------------+----------+
| {} | https://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ1.xml | 2024-03-28T00:10:00.074Z | monthly | 1.0 |
| {} | https://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ2.xml | 2023-06-20T23:44:00.215Z | monthly | 1.0 |
| {} | https://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ3.xml | 2023-07-03T14:32:01.529Z | monthly | 1.0 |
| {} | https://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ4.xml | 2023-08-29T19:17:07.501Z | monthly | 1.0 |
| {} | https://www.govinfo.gov/bulkdata/PLAW/118/public/PLAW-118publ5.xml | 2023-12-11T20:04:05.659Z | monthly | 1.0 |
+------------+--------------------------------------------------------------------+--------------------------+------------+----------+
5 rows selected (0.151 seconds)
```
No such luck with a [MODS file](https://www.govinfo.gov/metadata/pkg/PLAW-106publ246/mods.xml) though
```sql
apache drill> select * from dfs.`/Users/wm/Desktop/mods.xml`;
+------------+------+----------------+-------+----------+-----------+-----------+------------+---------------------+----------------+------------+------------+-----------------+-----------------+--------------------+---------------+---------+---------+-------+----------+
| attributes | name | typeOfResource | genre | language | extension | titleInfo | originInfo | physicalDescription | classification | identifier | recordInfo | accessCondition | isAppropriation | legislativeHistory | congCommittee | section | chapter | pages | location |
+------------+------+----------------+-------+----------+-----------+-----------+------------+---------------------+----------------+------------+------------+-----------------+-----------------+--------------------+---------------+---------+---------+-------+----------+
No rows selected (0.147 seconds)
```
Happy to add this as comment on #2908, but wanted to comment here in case there's something I'm missing as a brand new Drill user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org