CGSpace Notes

2018-02-01

• Peter gave feedback on the dc.rights proof of concept that I had sent him last week
• We don’t need to distinguish between internal and external works, so that makes it just a simple list
• Yesterday I figured out how to monitor DSpace sessions using JMX
• I copied the logic in the jmx_tomcat_dbpools provided by Ubuntu’s munin-plugins-java package and used the stuff I discovered about JMX in 2018-01

2018-01-02

• Uptime Robot noticed that CGSpace went down and up a few times last night, for a few minutes each time
• I didn’t get any load alerts from Linode and the REST and XMLUI logs don’t show anything out of the ordinary
• The nginx logs show HTTP 200s until 02/Jan/2018:11:27:17 +0000 when Uptime Robot got an HTTP 500
• In dspace.log around that time I see many errors like “Client closed the connection before file download was complete”
• And just before that I see this:

Caused by: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [http-bio-127.0.0.1-8443-exec-980] Timeout: Pool empty. Unable to fetch a connection in 5 seconds, none available[size:50; busy:50; idle:0; lastwait:5000].

• Ah hah! So the pool was actually empty!
• I need to increase that, let’s try to bump it up from 50 to 75
• After that one client got an HTTP 499 but then the rest were HTTP 200, so I don’t know what the hell Uptime Robot saw
• I notice this error quite a few times in dspace.log:

2018-01-02 01:21:19,137 ERROR org.dspace.app.xmlui.aspect.discovery.SidebarFacetsTransformer @ Error while searching for sidebar facets
org.dspace.discovery.SearchServiceException: org.apache.solr.search.SyntaxError: Cannot parse 'dateIssued_keyword:[1976+TO+1979]': Encountered " "]" "] "" at line 1, column 32.

• And there are many of these errors every day for the past month:

$ grep -c "Error while searching for sidebar facets" dspace.log.*
dspace.log.2017-11-21:4
dspace.log.2017-11-22:1
dspace.log.2017-11-23:4
dspace.log.2017-11-24:11
dspace.log.2017-11-25:0
dspace.log.2017-11-26:1
dspace.log.2017-11-27:7
dspace.log.2017-11-28:21
dspace.log.2017-11-29:31
dspace.log.2017-11-30:15
dspace.log.2017-12-01:15
dspace.log.2017-12-02:20
dspace.log.2017-12-03:38
dspace.log.2017-12-04:65
dspace.log.2017-12-05:43
dspace.log.2017-12-06:72
dspace.log.2017-12-07:27
dspace.log.2017-12-08:15
dspace.log.2017-12-09:29
dspace.log.2017-12-10:35
dspace.log.2017-12-11:20
dspace.log.2017-12-12:44
dspace.log.2017-12-13:36
dspace.log.2017-12-14:59
dspace.log.2017-12-15:104
dspace.log.2017-12-16:53
dspace.log.2017-12-17:66
dspace.log.2017-12-18:83
dspace.log.2017-12-19:101
dspace.log.2017-12-20:74
dspace.log.2017-12-21:55
dspace.log.2017-12-22:66
dspace.log.2017-12-23:50
dspace.log.2017-12-24:85
dspace.log.2017-12-25:62
dspace.log.2017-12-26:49
dspace.log.2017-12-27:30
dspace.log.2017-12-28:54
dspace.log.2017-12-29:68
dspace.log.2017-12-30:89
dspace.log.2017-12-31:53
dspace.log.2018-01-01:45
dspace.log.2018-01-02:34

• Danny wrote to ask for help renewing the wildcard ilri.org certificate and I advised that we should probably use Let’s Encrypt if it’s just a handful of domains

2017-12-01

• Uptime Robot noticed that CGSpace went down
• The logs say “Timeout waiting for idle object”
• PostgreSQL activity says there are 115 connections currently
• The list of connections to XMLUI and REST API for today:

2017-11-01

• The CORE developers responded to say they are looking into their bot not respecting our robots.txt

2017-11-02

• Today there have been no hits by CORE and no alerts from Linode (coincidence?)

# grep -c "CORE" /var/log/nginx/access.log
0

• Generate list of authors on CGSpace for Peter to go through and correct:

dspace=# \copy (select distinct text_value, count(*) as count from metadatavalue where metadata_field_id = (select metadata_field_id from metadatafieldregistry where element = 'contributor' and qualifier = 'author') AND resource_type_id = 2 group by text_value order by count desc) to /tmp/authors.csv with csv;
COPY 54701

2017-10-01

• Peter emailed to point out that many items in the ILRI archive collection have multiple handles:

http://hdl.handle.net/10568/78495||http://hdl.handle.net/10568/79336

• There appears to be a pattern but I’ll have to look a bit closer and try to clean them up automatically, either in SQL or in OpenRefine
• Add Katherine Lutz to the groups for content submission and edit steps of the CGIAR System collections

Rough notes for importing the CGIAR Library content. It was decided that this content would go to a new top-level community called CGIAR System Organization.

2017-09-06

• Linode sent an alert that CGSpace (linode18) was using 261% CPU for the past two hours

2017-09-07

• Ask Sisay to clean up the WLE approvers a bit, as Marianne’s user account is both in the approvers step as well as the group

2017-08-01

• Linode sent an alert that CGSpace (linode18) was using 350% CPU for the past two hours
• I looked in the Activity pane of the Admin Control Panel and it seems that Google, Baidu, Yahoo, and Bing are all crawling with massive numbers of bots concurrently (~100 total, mostly Baidu and Google)
• The good thing is that, according to dspace.log.2017-08-01, they are all using the same Tomcat session
• This means our Tomcat Crawler Session Valve is working
• But many of the bots are browsing dynamic URLs like:
  • /handle/10568/3353/discover
  • /handle/10568/16510/browse
• The robots.txt only blocks the top-level /discover and /browse URLs… we will need to find a way to forbid them from accessing these!
• Relevant issue from DSpace Jira (semi resolved in DSpace 6.0): https://jira.duraspace.org/browse/DS-2962
• It turns out that we’re already adding the X-Robots-Tag "none" HTTP header, but this only forbids the search engine from indexing the page, not crawling it!
• Also, the bot has to successfully browse the page first so it can receive the HTTP header…
• We might actually have to block these requests with HTTP 403 depending on the user agent
• Abenet pointed out that the CGIAR Library Historical Archive collection I sent July 20th only had ~100 entries, instead of 2415
• This was due to newline characters in the dc.description.abstract column, which caused OpenRefine to choke when exporting the CSV
• I exported a new CSV from the collection on DSpace Test and then manually removed the characters in vim using g/^$/d
• Then I cleaned up the author authorities and HTML characters in OpenRefine and sent the file back to Abenet

2017-07-01

• Run system updates and reboot DSpace Test

2017-07-04

• Merge changes for WLE Phase II theme rename (#329)
• Looking at extracting the metadata registries from ICARDA’s MEL DSpace database so we can compare fields with CGSpace
• We can use PostgreSQL’s extended output format (-x) plus sed to format the output into quasi XML:

2017-06-01

• After discussion with WLE and CGSpace content people, we decided to just add one metadata field for the WLE Research Themes
• The cg.identifier.wletheme field will be used for both Phase I and Phase II Research Themes
• Then we’ll create a new sub-community for Phase II and create collections for the research themes there
• The current “Research Themes” community will be renamed to “WLE Phase I Research Themes”
• Tagged all items in the current Phase I collections with their appropriate themes
• Create pull request to add Phase II research themes to the submission form: #328
• Add cg.subject.system to CGSpace metadata registry, for subject from the upcoming CGIAR Library migration

2017-06-04

• After adding cg.identifier.wletheme to 1106 WLE items I can see the field on XMLUI but not in REST!
• Strangely it happens on DSpace Test AND on CGSpace!
• I tried to re-index Discovery but it didn’t fix it
• Run all system updates on DSpace Test and reboot the server
• After rebooting the server (and therefore restarting Tomcat) the new metadata field is available
• I’ve sent a message to the dspace-tech mailing list to ask if this is a bug and whether I should file a Jira ticket

2016-06-05

• Rename WLE’s “Research Themes” sub-community to “WLE Phase I Research Themes” on DSpace Test so Macaroni Bros can continue their testing
• Macaroni Bros tested it and said it’s fine, so I renamed it on CGSpace as well
• Working on how to automate the extraction of the CIAT Book chapters, doing some magic in OpenRefine to extract page from–to from cg.identifier.url and dc.format.extent, respectively:
  • cg.identifier.url: value.split("page=", "")[1]
  • dc.format.extent: value.replace("p. ", "").split("-")[1].toNumber() - value.replace("p. ", "").split("-")[0].toNumber()
• Finally, after some filtering to see which small outliers there were (based on dc.format.extent using “p. 1-14” vs “29 p.”), create a new column with last page number:
  • cells["dc.page.from"].value.toNumber() + cells["dc.format.pages"].value.toNumber()
• Then create a new, unique file name to be used in the output, based on a SHA1 of the dc.title and with a description:
  • dc.page.to: value.split(" ")[0].replace(",","").toLowercase() + "-" + sha1(value).get(1,9) + ".pdf__description:" + cells["dc.type"].value
• Start processing 769 records after filtering the following (there are another 159 records that have some other format, or for example they have their own PDF which I will process later), using a modified generate-thumbnails.py script to read certain fields and then pass to GhostScript:
  • cg.identifier.url: value.contains("page=")
  • dc.format.extent: or(value.contains("p. "),value.contains(" p."))
  • Command like: $ gs -dNOPAUSE -dBATCH -dFirstPage=14 -dLastPage=27 -sDEVICE=pdfwrite -sOutputFile=beans.pdf -f 12605-1.pdf
• 17 of the items have issues with incorrect page number ranges, and upon closer inspection they do not appear in the referenced PDF
• I’ve flagged them and proceeded without them (752 total) on DSpace Test:

$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" [dspace]/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/93843 --source /home/aorth/src/CIAT-Books/SimpleArchiveFormat/ --mapfile=/tmp/ciat-books.map &> /tmp/ciat-books.log

• I went and did some basic sanity checks on the remaining items in the CIAT Book Chapters and decided they are mostly fine (except one duplicate and the flagged ones), so I imported them to DSpace Test too (162 items)
• Total items in CIAT Book Chapters is 914, with the others being flagged for some reason, and we should send that back to CIAT
• Restart Tomcat on CGSpace so that the cg.identifier.wletheme field is available on REST API for Macaroni Bros

2017-06-07

• Testing Atmire’s patch for the CUA Workflow Statistics again
• Still doesn’t seem to give results I’d expect, like there are no results for Maria Garruccio, or for the ILRI community!
• Then I’ll file an update to the issue on Atmire’s tracker
• Created a new branch with just the relevant changes, so I can send it to them
• One thing I noticed is that there is a failed database migration related to CUA:

+----------------+----------------------------+---------------------+---------+
| Version        | Description                | Installed on        | State   |
+----------------+----------------------------+---------------------+---------+
| 1.1            | Initial DSpace 1.1 databas |                     | PreInit |
| 1.2            | Upgrade to DSpace 1.2 sche |                     | PreInit |
| 1.3            | Upgrade to DSpace 1.3 sche |                     | PreInit |
| 1.3.9          | Drop constraint for DSpace |                     | PreInit |
| 1.4            | Upgrade to DSpace 1.4 sche |                     | PreInit |
| 1.5            | Upgrade to DSpace 1.5 sche |                     | PreInit |
| 1.5.9          | Drop constraint for DSpace |                     | PreInit |
| 1.6            | Upgrade to DSpace 1.6 sche |                     | PreInit |
| 1.7            | Upgrade to DSpace 1.7 sche |                     | PreInit |
| 1.8            | Upgrade to DSpace 1.8 sche |                     | PreInit |
| 3.0            | Upgrade to DSpace 3.x sche |                     | PreInit |
| 4.0            | Initializing from DSpace 4 | 2015-11-20 12:42:52 | Success |
| 5.0.2014.08.08 | DS-1945 Helpdesk Request a | 2015-11-20 12:42:53 | Success |
| 5.0.2014.09.25 | DS 1582 Metadata For All O | 2015-11-20 12:42:55 | Success |
| 5.0.2014.09.26 | DS-1582 Metadata For All O | 2015-11-20 12:42:55 | Success |
| 5.0.2015.01.27 | MigrateAtmireExtraMetadata | 2015-11-20 12:43:29 | Success |
| 5.0.2017.04.28 | CUA eperson metadata migra | 2017-06-07 11:07:28 | OutOrde |
| 5.5.2015.12.03 | Atmire CUA 4 migration     | 2016-11-27 06:39:05 | OutOrde |
| 5.5.2015.12.03 | Atmire MQM migration       | 2016-11-27 06:39:06 | OutOrde |
| 5.6.2016.08.08 | CUA emailreport migration  | 2017-01-29 11:18:56 | OutOrde |
+----------------+----------------------------+---------------------+---------+

• Merge the pull request for WLE Phase II themes

2017-06-18

• Redeploy CGSpace with latest changes from 5_x-prod, run system updates, and reboot the server
• Continue working on ansible infrastructure changes for CGIAR Library

2017-06-20

• Import Abenet and Peter’s changes to the CGIAR Library CRP community
• Due to them using Windows and renaming some columns there were formatting, encoding, and duplicate metadata value issues
• I had to remove some fields from the CSV and rename some back to, ie, dc.subject[en_US] just so DSpace would detect changes properly
• Now it looks much better: https://dspacetest.cgiar.org/handle/10947/2517
• Removing the HTML tags and HTML/XML entities using the following GREL:
  • replace(value,/<\/?\w+((\s+\w+(\s*=\s*(?:".*?"|'.*?'|[^'">\s]+))?)+\s*|\s*)\/?>/,'')
  • value.unescape("html").unescape("xml")
• Finally import 914 CIAT Book Chapters to CGSpace in two batches:

$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" [dspace]/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/35701 --source /home/aorth/CIAT-Books/SimpleArchiveFormat/ --mapfile=/tmp/ciat-books.map &> /tmp/ciat-books.log
$ JAVA_OPTS="-Xmx1024m -Dfile.encoding=UTF-8" [dspace]/bin/dspace import --add --eperson=aorth@mjanja.ch --collection=10568/35701 --source /home/aorth/CIAT-Books/SimpleArchiveFormat/ --mapfile=/tmp/ciat-books2.map &> /tmp/ciat-books2.log

2017-06-25

• WLE has said that one of their Phase II research themes is being renamed from Regenerating Degraded Landscapes to Restoring Degraded Landscapes
• Pull request with the changes to input-forms.xml: #329
• As of now it doesn’t look like there are any items using this research theme so we don’t need to do any updates:

dspace=# select text_value from metadatavalue where resource_type_id=2 and metadata_field_id=237 and text_value like 'Regenerating Degraded Landscapes%';
 text_value
------------
(0 rows)

• Marianne from WLE asked if they can have both Phase I and II research themes together in the item submission form
• Perhaps we can add them together in the same question for cg.identifier.wletheme

2017-06-30

• CGSpace went down briefly, I see lots of these errors in the dspace logs:

Java stacktrace: java.util.NoSuchElementException: Timeout waiting for idle object

• After looking at the Tomcat logs, Munin graphs, and PostgreSQL connection stats, it seems there is just a high load
• Might be a good time to adjust DSpace’s database connection settings, like I first mentioned in April, 2017 after reading the 2017-04 DCAT comments
• I’ve adjusted the following in CGSpace’s config:
  • db.maxconnections 30→70 (the default PostgreSQL config allows 100 connections, so DSpace’s default of 30 is quite low)
  • db.maxwait 5000→10000
  • db.maxidle 8→20 (DSpace default is -1, unlimited, but we had set it to 8 earlier)
• We will need to adjust this again (as well as the pg_hba.conf settings) when we deploy tsega’s REST API
• Whip up a test for Marianne of WLE to be able to show both their Phase I and II research themes in the CGSpace item submission form: