Raptorlicious: May 2017

Monday, May 29, 2017

Status of WoN on Linode

It looks to me that CPU usage is the limiting factor ( maybe >1.5 GHz processor?). Linode seems to fare better, although I will need to do a `docker stat` check there as well. I also tried Windows 10 with Amazon Workspaces, but the docker-compose script crashed immediately. My goal is to understand networking between the docker containers amongst themselves, the docker containers with the host system, and the network configuration of Linode such that specified ports are publicly accessible.

Sunday, May 28, 2017

self study links to learn about networking for Docker

https://docs.docker.com/engine/userguide/networking/

Keywords:
https://en.wikipedia.org/wiki/NAT_traversal
https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing

Books:
Networking bible / Barrie Sosinsky.

Tutorial (that involves setting up a webserver):
https://serversforhackers.com/getting-started-with-docker

Saturday, May 27, 2017

I'm confused about how Docker handles Hostnames

https://docs.docker.com/engine/userguide/networking/default_network/configure-dns/

Perhaps I could try the IP address of the machine hosting docker as well?

Friday, May 26, 2017

Steps taken to get the Docker Web of Needs (cont.) Running

Brian Chirgwin's May 20th Code Norman Docker Presentation:

https://docs.google.com/presentation/d/1ZuJjcnpPg5wnW9T52u_f3skM5CTxT402T3uBajuyxNc/edit#slide=id.g1c8951d7f5_0_3

Docker Practice site mentioned by Brian: http://training.play-with-docker.com/

Docker Hosting site mentioned by Brian: http://linode.com

Other hosting site mentioned by someone else: Amazon Workspace

Stats for RAM usage (inspired by conversation):

https://docs.docker.com/engine/reference/commandline/stats/

-------------------------------------------------------

I was not breaking 1GB of RAM, but curiously an image was using 142% of my CPU.

Sometimes docker stats did not show anything when performance was very poor.

---------------------------------------------------

My CPU: lscpu | grep MHz
CPU MHz: 1333.000

(https://askubuntu.com/questions/218567/any-way-to-check-the-clock-speed-of-my-processor)

sudo dmidecode -t processor | grep Speed
[sudo] password for brent:
Max Speed: 2000 MHz
Current Speed: 1000 MHz

For dynamic watching notice that everything goes to the maximum when running WoN:

watch -n1 "lscpu | grep 'CPU MHz' | awk '{print $1}'"

sudo watch -n 1 cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq

-------------------------------------------------

Trying Linode would give me:

Linode - Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz

(https://www.speakwp.com/reviews/comparing-linode-digital-ocean-5-vps-hosting/) 

--------------------------------------

Setting up a Linode Server:
https://www.distelli.com/docs/tutorials/how-to-set-up-a-linode-server/

Embrace the Docker Revolution (with Linode): https://medium.com/linode-cube/embrace-the-docker-revolution-in-3-easy-steps-ccd4ad077091

Thursday, May 25, 2017

Steps taken to get the Docker Web of Needs Swarm Running

**Actually, it does not seem to be a swam. it is running on one machine, but docker-compose is being used.

Beginning with https://github.com/researchstudio-sat/webofneeds/tree/master/webofneeds/won-docker I followed the steps:

(1) Install docker
Start with https://docs.docker.com/get-started/

https://docs.docker.com/engine/installation/linux/debian/#install-using-the-repository

(2) Install docker-compose
https://docs.docker.com/compose/install/
https://docs.docker.com/compose/overview/
and follow the tutorial
https://docs.docker.com/compose/gettingstarted/

(3) Create base_folder (mkdir data), .env file (see below), and docker-compose file (wget https://raw.githubusercontent.com/researchstudio-sat/webofneeds/master/webofneeds/won-docker/deploy/local/docker-compose.yml)

~/Documents/dockerexperiments/webofneeds$ ls -a
. .. data docker-compose.yml .env

~/Documents/dockerexperiments/webofneeds$ cat .env

deploy_host=localhost
base_folder=./data/

edit: see: https://docs.docker.com/compose/env-file/

(4) Run docker compose in the same directory without the detached option to enable debugging:
sudo docker-compose up (also "sudo service tomcat7 stop" and "sudo service mongodb stop" to prevent port conflicts)

(5) There is an error when trying to create a resource over https on localhost. Here are some instructions that may help set it up:
https://docs.docker.com/engine/security/https/

(6) Investigate the server cert:
~/Documents/dockerexperiments/webofneeds/data/won-server-certs$ openssl x509 -noout -text -in t-cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 9744456803442286009 (0x873b43d59a50ddb9)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=localhost
        Validity
            Not Before: May 25 00:12:13 2017 GMT
            Not After : May 25 00:12:14 2018 GMT
        Subject: CN=localhost
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:f5:a7:56:1a:c6:91:dc:3e:ed:82:ec:bc:73:87:
                    89:34:fc:e7:63:d8:e3:b6:21:0b:76:db:44:f8:6d:
                    5c:e7:e1:64:2b:48:76:a1:2e:b2:fe:28:dd:8b:3f:
                    8c:4c:1d:42:ca:bf:92:c0:90:fe:05:a6:db:2a:d6:
                    17:86:dd:5d:76:a0:86:7f:c2:ec:32:85:0a:91:d3:
                    7d:52:bf:43:bf:c1:84:8b:e0:30:e6:08:39:1c:c2:
                    63:3c:49:c2:11:8b:2e:b9:f9:39:78:40:59:4c:a2:
                    d7:b5:f4:c8:5b:68:72:72:51:f1:2d:8f:e7:0d:f5:
                    de:f7:38:42:4a:13:b0:23:c4:58:61:bc:1a:c0:b0:
                    09:38:4b:aa:06:76:b2:67:24:80:b2:71:81:bd:5e:
                    55:e4:6c:86:14:62:71:e5:e0:c8:97:f2:59:17:41:
                    b1:78:44:49:8c:28:bb:ab:9c:72:e1:7b:ec:7c:9b:
                    b5:31:da:66:18:8e:9e:61:89:a8:29:79:84:89:a2:
                    4f:dc:c0:39:4e:a8:e3:35:e1:0e:b8:2f:26:c8:2a:
                    7a:ad:11:23:25:5b:39:cc:81:bd:4b:66:b7:aa:31:
                    1c:69:79:6f:e6:03:fe:c0:eb:c9:98:6a:a6:32:3c:
                    b8:3e:c3:2b:f4:6b:34:66:6c:ae:ba:8f:b0:ef:23:
                    c0:b1
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                36:B7:33:A7:6D:AC:CE:1C:99:51:84:20:64:F4:A2:62:98:97:32:71
            X509v3 Authority Key Identifier:
                keyid:36:B7:33:A7:6D:AC:CE:1C:99:51:84:20:64:F4:A2:62:98:97:32:71

            X509v3 Basic Constraints:
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         be:73:e7:58:dd:78:38:36:56:14:90:bf:da:f1:ca:1f:1e:32:
         bf:92:8f:e7:ee:16:48:38:13:83:8d:ce:d2:ec:fd:45:9b:31:
         eb:b2:07:d0:b6:bf:5c:78:32:89:a2:e1:0c:18:ce:d0:19:ee:
         ae:75:85:6f:da:a5:81:d6:f2:aa:01:b1:65:33:ce:9d:ec:99:
         9f:3a:69:3f:66:68:c1:39:27:5f:23:ae:26:9b:60:d7:ef:48:
         74:41:b0:d2:0c:c9:fc:2c:7a:70:35:75:70:6f:71:64:72:78:
         d9:b1:b9:fb:f3:4e:45:a0:5c:83:b0:c4:35:25:7a:41:b3:4a:
         21:34:67:2d:af:28:1f:66:64:42:ab:e5:46:30:4a:6f:8e:d9:
         5a:62:bf:04:d5:fb:e7:3a:ca:86:60:35:9e:93:b3:94:89:81:
         d0:6a:5f:c9:80:27:c2:b2:88:54:a8:61:c9:b6:47:07:c0:3b:
         91:26:8e:7f:c7:62:44:48:a9:5b:30:96:27:22:1f:9c:ad:dd:
         30:42:b2:a7:71:39:2b:30:85:7b:0f:f4:8a:36:1a:ec:75:96:
         6e:53:c4:2e:f5:28:2c:57:de:e8:56:82:84:99:e1:1b:f0:d3:
         1d:6b:37:d6:53:da:e6:74:00:bb:69:f8:64:ba:83:b8:5e:49:
         7f:91:f4:60

========================
other interesting things
======================================

Interesting part of the log:
owner_1            | 25-May-2017 19:35:01.759 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

Interesting part of the log:
java.util.concurrent.TimeoutException

Interesting part of the log:
Connection refused:

Interesting part of the log:
blue hard drive light on...perhaps the RAM is low and SWAP space is being used, perhaps there are memory leaks

Fix the certificate due to https://localhost:8890 unable to create resource errors?:

/home/brent/Documents/dockerexperiments/webofneeds/data/won-server-certs# cp *.pem /home/brent/.docker/

(will this really work? The "Protect the Docker daemon socket" has a secure by default section that says to set the DOCKER_HOST and DOCKER_TLS_VERIFY variables (setting DOCKER_TLS_VERIFY=1 in the .env file causes an error))

=================================

Edit 6/02/2016: On Linode (http://linode.com) I used the 8192 standard instance [1]. (This provides greater than a ~1.3 GHz CPU). Then I pointed the public ip of the Linode instance at a domain (I used a subdomain from https://freedns.afraid.org). The environment variable deploy_host in the .env file was set to the subdomain (it seems it cannot be an IP address or the docker container name). I only went to (3).

I did not need to do any special network configuration. I'm guessing that the docker-compose.yml took care of opening the necessary ports and linking docker containers.

[1] Linode 8192
8

	GB RAM
4	CPU Cores (Intel® Xeon® processor E5 --- 2.3 GHz each??)
96	GB SSD Storage
4	TB Transfer
40	Gbps Network In
1000	Mbps Network Out

$40/mo or (.06/hr)

Friday, May 19, 2017

PowerAqua test May 19th

<?xml version="1.0" encoding="UTF-8"?>
<CONFIGURATION>
<WATSON>false</WATSON>
<POWERMAP>false</POWERMAP>
<VIRTUOSO>false</VIRTUOSO>
<REMOTESPARQLVIRTUOSO>false</REMOTESPARQLVIRTUOSO>
<ONTOLOGY_INDEX_DB>jdbc:mysql://127.0.0.1:3306/ontologyindexrelation</ONTOLOGY_INDEX_DB>
<ONTOLOGY_INDEX_DB_LOGIN>root</ONTOLOGY_INDEX_DB_LOGIN>
<ONTOLOGY_INDEX_DB_PASSWORD>password</ONTOLOGY_INDEX_DB_PASSWORD>
<INDEX_GLOBAL_PATH>./LuceneIndexes/</INDEX_GLOBAL_PATH>
<INDEX_INFO_FOLDER>./indexListInformation/indexbrentweb/</INDEX_INFO_FOLDER>

running: http://localhost:8080/poweraqua gives :

no error...

Clicking the ask button (with a string in the blank) gives:

java.lang.NullPointerException
 poweraqua.LinguisticComponent.GateChunker.ExecuteSerialController(GateChunker.java:322)
 poweraqua.LinguisticComponent.GateChunker.ParseQuestion(GateChunker.java:445)
 poweraqua.LinguisticComponent.LinguisticComponent.parseQuestion(LinguisticComponent.java:114)
 org.apache.jsp.jsp.answer_jsp._jspService(answer_jsp.java:167)
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
 org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
 org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395)
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

Monday, May 15, 2017

Change Log for PowerAqua (may 15th)

In /var/lib/tomcat7/webapps/poweraqua, change service_properties.xml from:

<?xml version="1.0" encoding="UTF-8"?><CONFIGURATION><PLUGIN_MANAGER>/var/lib/tomcat7/webapps/poweraqua/WEB-INF/aquaplugins</PLUGIN_MANAGER><REPOSITORY><SERVER>127.0.0.1:8080/openrdf-sesame</SERVER><PROXY></PROXY><PORT></PORT><LOGIN></LOGIN><PASSWORD></PASSWORD><PLUGIN_TYPE>sesame2</PLUGIN_TYPE><REPOSITORY_NAME>isp</REPOSITORY_NAME><TYPE>OWL</TYPE></REPOSITORY></CONFIGURATION>

to:

<?xml version="1.0" encoding="UTF-8"?><CONFIGURATION><PLUGIN_MANAGER>/var/lib/tomcat7/webapps/poweraqua/WEB-INF/aquaplugins</PLUGIN_MANAGER><REPOSITORY><SERVER>http://127.0.0.1:8080/openrdf-sesame</SERVER><PROXY></PROXY><PORT></PORT><LOGIN></LOGIN><PASSWORD></PASSWORD><PLUGIN_TYPE>sesame2</PLUGIN_TYPE><REPOSITORY_NAME>isp</REPOSITORY_NAME><TYPE>OWL</TYPE></REPOSITORY></CONFIGURATION>

(note, I might need to use localhost instead of 127.0.0.1)

Also in /var/lib/tomcat7/webapps/poweraqua/indexListInformation/indexbrentweb, change service_properties.xml from:

<?xml version="1.0" encoding="UTF-8"?>
<CONFIGURATION>
<PLUGIN_MANAGER>/var/lib/tomcat7/webapps/poweraqua/WEB-INF/aquaplugins</PLUGIN_MANAGER>
<REPOSITORY>
    <SERVER>127.0.0.1:8080/openrdf-sesame</SERVER>
    <PROXY></PROXY>
    <PORT></PORT>
    <LOGIN></LOGIN>
    <PASSWORD></PASSWORD>
    <PLUGIN_TYPE>sesame2</PLUGIN_TYPE>
    <REPOSITORY_NAME>isp</REPOSITORY_NAME>
    <TYPE>OWL</TYPE>
</REPOSITORY>
</CONFIGURATION>

to:

<?xml version="1.0" encoding="UTF-8"?>
<CONFIGURATION>
<PLUGIN_MANAGER>/var/lib/tomcat7/webapps/poweraqua/WEB-INF/aquaplugins</PLUGIN_MANAGER>
<REPOSITORY>
    <SERVER>http://127.0.0.1:8080/openrdf-sesame</SERVER>
    <PROXY></PROXY>
    <PORT></PORT>
    <LOGIN></LOGIN>
    <PASSWORD></PASSWORD>
    <PLUGIN_TYPE>sesame2</PLUGIN_TYPE>
    <REPOSITORY_NAME>isp</REPOSITORY_NAME>
    <TYPE>OWL</TYPE>
</REPOSITORY>
</CONFIGURATION>

After running the shell script: /var/lib/tomcat7/webapps/poweraqua# ./createLuceneIndex.sh

the following log results:
https://gist.github.com/bshambaugh/d00017026e916d917030628c0ebec7b7

tl;dr: it helped a lot to put http in the server path before 127.0.0.1, so I avoid line 1152 "java.lang.IllegalStateException: unsupported protocol: '127.0.0.1'" in https://gist.github.com/bshambaugh/c06f3b2b784f47e5a6e123df2ab42fed

./createLuceneIndex.sh also produced the following changes to the MySQL table:

But ontologyindextable is empty :( .

Backup of the ontology on gist: https://gist.github.com/bshambaugh/04581f98faa875e87c96684566de7ad8

Loading PowerAqua at http://localhost:8080/poweraqua/jsp/init.jsp gives:

What I worked on Yesterday

I played with jQuery dialog boxes yesterday [1], thought about media queries to adjust the CSS grid layout, and read about scaling SVG documents [2] as well as button creation [3] so I can make my own scaleable [2] buttons.

[1] https://jqueryui.com/dialog/#modal-form
[2] https://css-tricks.com/scale-svg/
[3] https://www.youtube.com/watch?v=iOWzFqrdOVk

Raptorlicious