tag:blogger.com,1999:blog-185194812024-02-21T14:33:56.258+08:00techblog.simoncpu.comeW91IGhhdmUgbG9zdCB0aGUgZ2FtZTsgYWxzbywgYjAwYmllcyAoLlkuKQ==simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.comBlogger139125tag:blogger.com,1999:blog-18519481.post-87481007834471117242017-10-24T00:56:00.000+08:002017-10-24T01:00:24.618+08:00HOWTO access AWS resources inside a private VPCSorry, no screenshots. I just encountered this problem in my work, and creating screenshots will redo the entire thing. :)<br />
<br />
<b>Scenario</b><br />
Your AWS resources is inside a private VPC that's inaccessible from the outside world. In our specific case, we have Lambda functions located inside a VPC together with ElastiCache. I needed a way to load data into ElastiCache.<br />
<br />
<b>Solution (note: this is not be step by step, I'm just recreating the required components; there's probably a wizard for this or something).</b><br />
<br />
<ol>
<li>In the VPC page, create a VPC. Let's name this vpc-111</li>
<li>Create an Internet Gateway. Let's name this igw-12345.</li>
<li>Create a subnet for igw-12345 and in your Route Table, point 0.0.0.0/0 to igw-12345. Let's name this subnet subnet-2468.</li>
<li>Launch an EC2 instance and put it inside vpc-111. Enable public IP auto-assignment. Select subnet-2468 for your instance.</li>
<li>Launch your EC2 instance.</li>
<li>You can now access the AWS resources inside the instance or you can use <a href="https://www.ssh.com/ssh/tunneling/example" rel="nofollow">SSH tunneling</a> to access them from your workstation.</li>
<li>Enjoy!</li>
</ol>
simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-48251123101757030872017-07-22T07:12:00.000+08:002017-07-22T07:12:21.516+08:00How to let Apache Nutch index the results via ElasticSearch over REST API.In my work, we used Apache Nutch to fetch and parse some data from various websites. The data is pretty huge, so we need to index this on ElasticSearch. In my initial prototypes, I discovered that we can't use AWS ElasticSearch because AWS doesn't expose the native transport protocol; we can only talk to it via its RESTful API. In the official Nutch releases (v2.3.1 and v1.13, as of July 22, 2017), the plugin for ES doesn't support the RESTful API. When I checked the source code though, I was quite surprised that there's another plugin for this. Unfortunately, there's zero documentation on this, so I wish these notes would help others. Note: the URL regex config and HTTP options need to be set too, the IP addresses need to be whitelisted, etc... just leave a comment in case I miss something.<br />
<br />
OS used for this experiment: Ubuntu 16.04.2 LTS<br />
<div>
<br /></div>
<div>
<h1 data-usually-unique-id="052399270441109392976484">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Install Java</span></h1>
</div>
<div>
<code class="listtype-code listindent1 list-code1 lang-sql" spellcheck="false">add-apt-repository ppa:webupd8team/java</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-sql" spellcheck="false">apt-get update</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-sql" spellcheck="false">apt-get install oracle-java8-installer</code></div>
<div>
<h1 data-usually-unique-id="084437400116228630778889">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Install ant</span></h1>
</div>
<div>
<code class="listtype-code listindent1 list-code1 lang-swift" spellcheck="false">apt-get install ant</code></div>
<div>
<h1 data-usually-unique-id="183771974584587684309278">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Increase the ulimit settings</span></h1>
</div>
<div>
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">(TODO: these settings should survive a reboot; just edit /etc/sysctl.conf or something)</span></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-tex" spellcheck="false">ulimit -f unlimited \</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-tex" spellcheck="false"> -t unlimited \</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-tex" spellcheck="false"> -v unlimited \</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-tex" spellcheck="false"> -n 64000 \</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-tex" spellcheck="false"> -m unlimited \</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-tex" spellcheck="false"> -u 64000</code></div>
<div>
<h1 data-usually-unique-id="962268503660692758795363">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Install Apache Nutch</span></h1>
</div>
<div>
<h2 data-usually-unique-id="930838861353923749198833">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Download the source code</span></h2>
</div>
<div>
<code class="listtype-code listindent1 list-code1 lang-less" spellcheck="false">git clone https://github.com/apache/nutch.git</code></div>
<div>
<br /></div>
<div>
<h2 data-usually-unique-id="583792257026858633974442">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Build</span></h2>
</div>
<div>
<code class="listtype-code listindent1 list-code1" spellcheck="false">ant clean runtime</code></div>
<div>
<br /></div>
<div>
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Put this inside </span><span class="inline-code author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z"><span style="background-color: #f7f9fa; border: 1px solid rgba(208,212,217,0.5); color: #1b2733; font-family: monospace;">runtime/local/config/nutch-site.xml</span></span><span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">:</span></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><?xml version="1.0"?></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><configuration></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>http.agent.name</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>Spiderman</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>plugin.includes</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>protocol-(http|httpclient)|urlfilter-regex|index-(basic|more)|query-(basic|site|url|lang)|indexer-elastic|indexer-elastic-rest|nutch-extensionpoints|parse-(text|html|msexcel|msword|mspowerpoint|pdf)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)|parse-(html|tika|metatags)|index-(basic|anchor|more|metadata)</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>parser.character.encoding.default</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>utf-8</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<br /></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>elastic.rest.host</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>aws-elastic-search-endpoint.example.org</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>elastic.rest.port</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>443</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>elastic.rest.index</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>nutch</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>elastic.rest.type</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>doc</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>elastic.rest.https</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>true</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>elastic.rest.trustallhostnames</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>false</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"></configuration></code></div>
<div>
<br /></div>
<div>
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Make sure to whitelist your IP address first in AWS ElasticSearch’s policy settings. Yikes it's 7:11AM now... need to sleep. Hahaha... :p</span></div>
simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-36818686426810824912017-07-13T19:08:00.000+08:002017-07-13T19:08:42.524+08:00Setting up Apache Nutch v2.3.1 on Ubuntu with MongoDB and ElasticSearchThese are my notes for installing Apache Nutch v2.3.1 on Ubuntu in my work. We'll probably not use Nutch 2.3 because AWS ElasticSearch doesn't support the native transport protocol (it only supports the REST API, and it exposes it in a non-standard manner on port 80); thus we'll have a hard time indexing the data unless we maintain the servers ourselves. We can probably use AWS CloudSearch, but Nutch 2.3 doesn't have a plugin for it, unlike Nutch 1.13.<br />
<br />
Anyway, there are lots of similar HOWTOs out there, but this one points to specific versions and URLs so you can just copy and paste these to get your system up and running in a few minutes.<br />
<br />
<div>
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">OS used for this experiment: Ubuntu 16.04.2 LTS</span></div>
<div>
<h1 data-usually-unique-id="534171655531944940807608">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Make sure that Ubuntu is the latest version</span></h1>
</div>
<div>
<code class="listtype-code listindent1 list-code1 lang-sql" spellcheck="false">apt-get update</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-sql" spellcheck="false">apt-get dist-upgrade</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-sql" spellcheck="false">reboot #if necessary</code></div>
<div>
<br /></div>
<div>
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Updating to the latest version is needed to fix the glibc bug that affects MongoDB. It’s probably not required anymore, but it’s good to be safe.</span></div>
<div>
<h1 data-usually-unique-id="052399270441109392976484">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Install Java</span></h1>
</div>
<div>
<code class="listtype-code listindent1 list-code1 lang-sql" spellcheck="false">add-apt-repository ppa:webupd8team/java</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-sql" spellcheck="false">apt-get update</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-sql" spellcheck="false">apt-get install oracle-java8-installer</code></div>
<div>
<h1 data-usually-unique-id="084437400116228630778889">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Install ant</span></h1>
</div>
<div>
<code class="listtype-code listindent1 list-code1 lang-swift" spellcheck="false">apt-get install ant</code></div>
<div>
<h1 data-usually-unique-id="183771974584587684309278">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Increase the ulimit settings</span></h1>
</div>
<div>
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">(TODO: these settings should survive a reboot)</span></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-tex" spellcheck="false">ulimit -f unlimited \</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-tex" spellcheck="false"> <span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">-t unlimited \</span></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-tex" spellcheck="false"> <span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">-v unlimited \</span></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-tex" spellcheck="false"> <span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">-n 64000 \</span></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-tex" spellcheck="false"> <span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">-m unlimited \</span></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-tex" spellcheck="false"> <span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">-u 64000</span></code></div>
<div>
<h1 data-usually-unique-id="702896807681194543865067">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Install MongoDB</span></h1>
</div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false">apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 0C49F3730359A14518585931BC711F9BA15703C6</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false">echo "deb [ arch=amd64,arm64 ] http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.4.list</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false">apt-get update</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false">apt-get install mongodb-org</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false">service mongod start</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false">echo "use nutch" | mongo</code></div>
<div>
<h1 data-usually-unique-id="962268503660692758795363">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Install Apache Nutch</span></h1>
</div>
<div>
<h2 data-usually-unique-id="930838861353923749198833">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Download and extract</span></h2>
</div>
<div>
<code class="listtype-code listindent1 list-code1 lang-less" spellcheck="false">wget http://apache.cs.utah.edu/nutch/2.3.1/apache-nutch-2.3.1-src.tar.gz</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-less" spellcheck="false">tar xvf apache-nutch-2.3.1-src.tar.gz</code></div>
<div>
<h2 data-usually-unique-id="045561552065401177086474">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Edit the configs</span></h2>
</div>
<div>
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z"><b>Uncomment this line at </b></span><span class="inline-code author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z"><span style="background-color: #f7f9fa; border: 1px solid rgba(208 , 212 , 217 , 0.5); color: #1b2733; font-family: monospace;">apache-nutch-2.3.1/ivy.xml</span></span><span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">:</span></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><dependency org="org.apache.gora" name="gora-mongodb" rev="0.6.1" conf="*->default" /></code></div>
<div>
<br /></div>
<div>
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z"><b>Uncomment these lines at </b></span><span class="inline-code author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z"><span style="background-color: #f7f9fa; border: 1px solid rgba(208 , 212 , 217 , 0.5); color: #1b2733; font-family: monospace;">apache-nutch-2.3.1/runtime/local/conf/gora.properties</span></span><span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">:</span></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-cpp" spellcheck="false">gora.datastore.default=org.apache.gora.mongodb.store.MongoStore</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-cpp" spellcheck="false">gora.mongodb.override_hadoop_configuration=false</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-cpp" spellcheck="false">gora.mongodb.mapping.file=/gora-mongodb-mapping.xml</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-cpp" spellcheck="false">gora.mongodb.servers=localhost:27017</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-cpp" spellcheck="false">gora.mongodb.db=nutch</code></div>
<div>
<h2 data-usually-unique-id="920443057210785178874607">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Build Apache Nutch</span></h2>
</div>
<div>
<code class="listtype-code listindent1 list-code1 lang-bash" spellcheck="false">ant runtime #needs to be run in the top directory</code></div>
<div>
<h1 data-usually-unique-id="831870390601741116867718">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Install ElasticSearch</span></h1>
</div>
<div>
<span class="ace-all-bold-hthree"><span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z"><b><s>Setup ElasticSearch on AWS</s></b></span></span></div>
<div>
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z"><s>Create a new AWS ElasticSearch domain and whitelist the server(s) where Apache Nutch is installed.</s></span><span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z"> AWS ElasticSearch doesn’t support the native transport protocol</span><span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z s-lparen"> </span><span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z h-lparen">(</span><span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">port 9300). We may be able to use AWS </span><span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z"><i>if</i></span><span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z"> Nutch supports the HTTP REST protocol. Needs more research.</span></div>
<div>
<br /></div>
<div>
<span class="ace-all-bold-hthree"><span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z"><b>Manual Installation</b></span></span></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false"># It's possible to use the latest version, ES5, but you need to update the indexer.</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false">wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false">echo "deb http://packages.elastic.co/elasticsearch/1.7/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-1.7.list</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false">apt-get update && sudo apt-get install elasticsearch</code></div>
<div>
<br /></div>
<div>
<span class="ace-all-bold-hthree"><span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z"><b>Edit </b></span><span class="inline-code author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z"><b><span style="background-color: #f7f9fa; border: 1px solid rgba(208 , 212 , 217 , 0.5); color: #1b2733; font-family: monospace;">/etc/elasticsearch/elasticsearch.yaml</span></b></span></span><span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z"><b> and put this:</b></span></div>
<div>
<code class="listtype-code listindent1 list-code1" spellcheck="false">network.host: 127.0.0.1</code></div>
<div>
<code class="listtype-code listindent1 list-code1" spellcheck="false">cluster.name: nutch</code></div>
<div>
<code class="listtype-code listindent1 list-code1" spellcheck="false">node.name: nutch1</code></div>
<div>
<br /></div>
<div>
<span class="ace-all-bold-hthree"><span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z"><b>Finish up</b></span></span></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-sql" spellcheck="false">update-rc.d elasticsearch defaults</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-sql" spellcheck="false">/etc/init.d/elasticsearch restart</code></div>
<div>
<h2 data-usually-unique-id="808769248888788960960640">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Install Kibana (optional, didn't install this since curl works fine)</span></h2>
</div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false">echo "deb http://packages.elastic.co/kibana/4.4/debian stable main" | sudo tee -a /etc/apt/sources.list.d/kibana-4.4.x.list</code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-php" spellcheck="false">apt-get update && sudo apt-get install kibana</code></div>
<div>
<h2 data-usually-unique-id="932090986903786699008806">
<span class=" author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z">Put this inside </span><span class="inline-code author-d-iz88z86z86za0dz67zz78zz78zz74zz68zjz80zz71z9iz90z95lz87zchz67zz72zz67z1z80zz70zz65zz68z3z75zo4z83zz82zz86zvuz71zggz80zz65zgoz88z"><span style="background-color: #f7f9fa; border: 1px solid rgba(208 , 212 , 217 , 0.5); color: #1b2733; font-family: monospace;">apache-nutch-2.3.1/runtime/local/conf/nutch-site.xml</span></span></h2>
</div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><?xml version="1.0"?></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><?xml-stylesheet type="text/xsl" href="configuration.xsl"?></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><configuration></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>storage.data.store.class</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>org.apache.gora.mongodb.store.MongoStore</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <description>Default class for storing data</description></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>http.agent.name</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>Spiderman</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>plugin.includes</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>protocol-(http|httpclient)|urlfilter-regex|index-(basic|more)|query-(basic|site|url|lang)|indexer-elastic|nutch-extensionpoints|parse-(text|html|msexcel|msword|mspowerpoint|pdf)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)|parse-(html|tika|metatags)|index-(basic|anchor|more|metadata)</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>elastic.host</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>127.0.0.1</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>elastic.port</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>9300</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>elastic.cluster</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>nutch</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>elastic.index</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>nutch</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>parser.character.encoding.default</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>utf-8</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>http.content.limit</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>6553600</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>elastic.max.bulk.docs</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>2000</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <name>elastic.max.bulk.size</name></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> <value>2500500</value></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"> </property></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"><br /></code></div>
<div>
<code class="listtype-code listindent1 list-code1 lang-xml" spellcheck="false"></configuration></code></div>
simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-84988810400878890932017-07-05T17:30:00.000+08:002017-07-06T21:57:43.443+08:00Automatically scaling and deploying Server Density agent on AWS Elastic BeanstalkWe're using AWS ElasticBeanstalk in one of our projects at my work. EB is an abstraction on top of Elastic Load Balancing and EC2 (which is yet another abstraction on top of Xen, but I digress hehe...), and I really appreciate how it simplifies a lot of routine sysad tasks. EB allows us to scale the website up and down, depending on the traffic.<br />
<br />
One problem that we encountered during development was a memory leak from PHP. To prevent another downtime (or at least help predict when the servers are about to go offline), I decided to use <a href="https://www.serverdensity.com/">Server Density</a> to implement an early warning system that would give us a heads up when the server's memory is about to run out. Server Density is a pretty great service. We've been using it for quite some time now. Server Density works by letting you install an agent inside of your servers, and the agent will then continuously push metrics to their server. Server Density then collects all these logs and displays them into nice graphs. If a metric crosses the threshold in one of your conditions (i.e., if the CPU load is >= 90% or disk space is <= 5GB), Server Density then sends you notification via e-mail, SMS, and Slack.<br />
<br />
The Server Density agent can be installed during each EB deployment by executing this inside your deployment script at .ebextensions/:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">curl https://archive.serverdensity.com/agent-install.sh | bash -s -- -a <span style="color: red;">ACCOUNTNAME </span>-t <span style="color: red;">PUTYOURKEYHERE </span>-g <span style="color: red;">GroupNameForYourEB </span>-p amazon -i</span><br />
<br />
Determining the amount of free memory in Linux is somewhat tricky. I can't use the memory usage metric to trigger a notification because Linux always shows that nearly all of the memory is being used, even though in reality they're just being used as cache. To get a more reliable metric, I used swap space instead. The theory is that the instances are supposed to have enough RAM for the tasks; when memory runs out, Linux uses the swap space as a last resort. Thus, we can check if swap space is >= 1MB to trigger a notification.<br />
<br />
Next problem: since EB regularly deploys and terminates the instances, Server Density ends up monitoring servers that no longer exist. We needed a way to automatically stop Server Density from monitoring instances that have been terminated.<br />
<br />
Solution: I made a CloudWatch rule that triggers whenever instances are stopped or terminated. The events are then pushed to a Lambda function which calls Server Density's API to remove the monitoring.<br />
<br />
Here's the architecture that I came up with:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVexo-pMqky_-ssI-3q4Y-0-ac2F4EbrfPl9YYH_0zMWNMHHu7XWnQJMoWI1Oh6Z1l3wMx8AW9W8PCTaTxp8E9Je3XHKGGSu6FRPewOniHT7fL95GELnzidat3uYEQG661OM0J/s1600/sd-architecture.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="660" data-original-width="1160" height="363" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVexo-pMqky_-ssI-3q4Y-0-ac2F4EbrfPl9YYH_0zMWNMHHu7XWnQJMoWI1Oh6Z1l3wMx8AW9W8PCTaTxp8E9Je3XHKGGSu6FRPewOniHT7fL95GELnzidat3uYEQG661OM0J/s640/sd-architecture.png" width="640" /></a></div>
I think CloudWatch has a way to monitor the swap space, but the last time I checked, AWS SNS (a separate AWS service that sends notifications) can't send SMS messages to Philippine numbers so I can't wake up (not joking, unfortunately haha) whenever there are server problems.<br />
<br />
Update: Turns out that Linux' default swappiness value is 60, which means that it will use swap ahead of time even though around half (or 80%? the docs have conflicting calculations) of the RAM is still available. To avoid this situation, set the swappiness to 1. You can even set it to 0 if you want:<br />
<div style="text-align: center;">
<span style="font-family: Courier New, Courier, monospace;">sysctl vm.swappiness=1</span></div>
<br />simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-90771187773444887462017-04-27T17:12:00.003+08:002017-04-28T14:21:19.911+08:00How to run AWS Lambda on VPCI'm trying to run an AWS Lambda function inside a VPC because I need it to access ElastiCache. Problem: if you put a Lambda function inside a VPC, it loses Internet access. There are some few documentations online, but they are too complicated and they involve unnecessary steps. Here is the simplest way to run AWS Lambda inside VPC.<br />
<br />
1. Create a simple NodeJS function that connects to an external site via HTTP:<br />
<span style="font-family: "courier new" , "courier" , monospace;">'use strict';</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">const http = require('http');</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">exports.handler = (event, context, callback) => {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> http.get('http://www.google.com', res => {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> callback(null, `Success, with: ${res.statusCode}`);</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> }).on('error', err => callback(`Error with: ${err.message}`));</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">};</span><br />
<div>
<br /></div>
<div>
2. Run the above function without a VPC to verify that it's working correctly (i.e., it returns an HTTP 200).</div>
<div>
<br /></div>
<div>
3. In the AWS Console, go to the VPC page and click "Elastic IPs". Then, click the "Allocate new address" button and select the "VPC" scope.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh03WztPXzWE3CxNv5H6XONeMsfcKhha11jok2kXz1lTZrsHgudaOxofWx-e-S0rvEmUCHcE6sA_NPNRyUg6TwY9Cg-rEuB3ZOQLCtVDO_um65PlL4JBXhE7QAl13_NT-eJmHUi/s1600/Screenshot+at+2017-04-27+16-51-09.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh03WztPXzWE3CxNv5H6XONeMsfcKhha11jok2kXz1lTZrsHgudaOxofWx-e-S0rvEmUCHcE6sA_NPNRyUg6TwY9Cg-rEuB3ZOQLCtVDO_um65PlL4JBXhE7QAl13_NT-eJmHUi/s1600/Screenshot+at+2017-04-27+16-51-09.png" /></a></div>
<div>
<br /></div>
<div>
4. Next, go to the VPC Dashboard and click the "Start VPC Wizard" button.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhG5yiiomv74aK7iqBCNGWo5lY4Yr1apiiSk2TWmQ-cLxokMHKVWkWONwvDntRPD8gK-bXoBWTXUHVu8RvXhWLNkjzhaCb_dvWOyJv2iBvrQkUQvr_UGsaLBkXF_gTA5GLcL2fC/s1600/Screenshot+at+2017-04-27+16-47-45.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhG5yiiomv74aK7iqBCNGWo5lY4Yr1apiiSk2TWmQ-cLxokMHKVWkWONwvDntRPD8gK-bXoBWTXUHVu8RvXhWLNkjzhaCb_dvWOyJv2iBvrQkUQvr_UGsaLBkXF_gTA5GLcL2fC/s1600/Screenshot+at+2017-04-27+16-47-45.png" /></a></div>
<div>
<br /></div>
<div>
5. Select VPC with Public and Private Subnets option.</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEUhXoiHYYr4N2J5IMnSROr3vWm4QPERKuDNkiO6wormQj-Olwo5XA9l-P9tAbPMBfldq11_KZU4bTlhHXZFgg67PV7MaK1dxA0R1jDSKWDhWjtYfga7SPW6lKmJlQmQPrqiRk/s1600/Screenshot+at+2017-04-27+16-49-10.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEUhXoiHYYr4N2J5IMnSROr3vWm4QPERKuDNkiO6wormQj-Olwo5XA9l-P9tAbPMBfldq11_KZU4bTlhHXZFgg67PV7MaK1dxA0R1jDSKWDhWjtYfga7SPW6lKmJlQmQPrqiRk/s1600/Screenshot+at+2017-04-27+16-49-10.png" /></a></div>
<div>
<br /></div>
<div>
6. In the next page, enter your "VPC name", and in the "Elastic IP Allocation ID" field, enter the Elastic IP that you created in Step 3. Click the "Create VPC" button.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheY5VB6FldVtVITfv9-RAX0_rnESfKadt3NPg8LJzF0m_pI78S8OO2dQa6qZopkymPBFPXTsOjQdhksmvwEYjvrtYDmGyXz3t3Y5Hr61_PqpfcN18f1jJDaxIwNphaHIzIWXh0/s1600/Screenshot+at+2017-04-28+14-16-53.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheY5VB6FldVtVITfv9-RAX0_rnESfKadt3NPg8LJzF0m_pI78S8OO2dQa6qZopkymPBFPXTsOjQdhksmvwEYjvrtYDmGyXz3t3Y5Hr61_PqpfcN18f1jJDaxIwNphaHIzIWXh0/s1600/Screenshot+at+2017-04-28+14-16-53.png" /></a></div>
<br /></div>
<div>
7. Finally, go back to the Lambda page and configure your function. Click the "Configuration" tab and go to the "Advanced settings section". Select the VPC that you created in Step 5, and Select the <i><span style="color: red;">private subnet</span></i> that you created. This is important; otherwise outgoing Internet connections won't work.</div>
<div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjQQ-wDlfXE99hngEniH3LDChqQMe7z5_S2sNKDqzg4_d9gT1F-eUqf2lgGnY_mgvSw0W2uxwttUwhzoZ0dRwDqctohjj030YiByYoQnPFSawDxeOSsGycbhXoKGrP04X0-Jcw/s1600/Screenshot+at+2017-04-27+17-05-31.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjQQ-wDlfXE99hngEniH3LDChqQMe7z5_S2sNKDqzg4_d9gT1F-eUqf2lgGnY_mgvSw0W2uxwttUwhzoZ0dRwDqctohjj030YiByYoQnPFSawDxeOSsGycbhXoKGrP04X0-Jcw/s1600/Screenshot+at+2017-04-27+17-05-31.png" /></a></div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
8. Click the "Save and test" button to test your setup. That's it! For a proper setup, use at least 2 subnets in different availability zones to run your function in high-availability mode.</div>
<div>
<br /></div>
<div>
Some SEO keywords to help other people: aws lambda run vpc nat gateway howto tutorial</div>
simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-17336209552906471952017-03-23T19:35:00.001+08:002017-03-24T15:36:32.494+08:00How to run arbitrary binary executables on AWS Lambda 6.10+After posting the hack yesterday, AWS coincidentally released the Node.js 6.10 platform. The old method from 0.10 now works:<br />
<br />
1. Create a bin/ directory inside your project directory.<br />
2. Statically compile your binary as much as possible. Put it inside your bin/ directory. Put any shared libraries that it needs inside the bin/ directory too.<br />
3. chmod +x bin/hello-world-binary<br />
4. Do something like:<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">'use strict';</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">const exec = require('child_process').exec;</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="color: red; font-family: "courier new" , "courier" , monospace;"><b>process.env['PATH'] = process.env['PATH'] + ':' + process.env['LAMBDA_TASK_ROOT'] + '/bin';</b></span><br />
<span style="color: red; font-family: "courier new" , "courier" , monospace;"><b>process.env['LD_LIBRARY_PATH'] = process.env['LAMBDA_TASK_ROOT'] + '/bin';</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">exports.handler = (event, context, callback) => {</span><br />
<span style="color: red; font-family: "courier new" , "courier" , monospace;"><b> const cmd = 'hello-world-binary';</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> const child = exec(cmd, (error) => {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> callback(error, 'Process complete!');</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> });</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> child.stdout.on('data', console.log);</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> child.stderr.on('data', console.error);</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">};</span><br />
<div>
<br /></div>
<div>
5. ZIP your project directory and upload to AWS Lambda. Enjoy!</div>
simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-13587419572397151362017-03-22T19:07:00.000+08:002017-03-24T15:37:56.065+08:00How to run arbitrary binary executables on AWS Lambda 4.3+I've been enjoying AWS Lambda in my work lately. I appreciate how we can create serverless architecture (or nearly serverless, anyway) on the AWS Platform. Yeah, it's fun to set up Linux and/or FreeBSD servers myself, but oftentimes, this "cloud" thingy makes perfect sense in many scenarios.<br />
<br />
Anyway, AWS Lambda has various constraints, such as not being able to install packages yourself. These constraints, according to their documentation, allows them to "perform operational and administrative activities on [our] behalf, including provisioning capacity, monitoring fleet health, applying security patches, deploying [our] code, and monitoring and logging [our] Lambda functions."<br />
<br />
OK, fine; but how do you deal with code that depends on an external binary? In previous versions of the Lambda platform (Node.js v0.10), you can do this by statically compiling your binary and by putting it inside the bin/ folder inside your project and, by doing something like:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">process.env['PATH'] = process.env['PATH'] + ':' + process.env['LAMBDA_TASK_ROOT'] + '/bin';</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">process.env['LD_LIBRARY_PATH'] = process.env['LAMBDA_TASK_ROOT'] + '/bin';</span><br />
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">exec('hello-world-binary', function () {});</span></div>
<br />
With the recent update, however, the above code no longer works. On AWS Lambda platform that uses Node.js v4.3+, we'd need to copy all the binaries to the /tmp folder, chmod +x it, and run it from there.<br />
<br />
I couldn't find a cleaner solution, so I just wrapped these in a Node.js package:<br />
<br />
<a href="https://www.npmjs.com/package/lambda_exec">https://www.npmjs.com/package/lambda_exec</a><br />
<br />
Enjoy!simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-27241256816976598622016-12-05T18:45:00.003+08:002016-12-05T19:11:44.185+08:00Disabling WoSign and StartCom from list of trusted CAsI was setting up an <a href="https://openvpn.net/">OpenVPN</a> server a while ago. To remove the browser warning on its WebGUI, I used a free certificate from StartSSL because it's more straightforward than <a href="https://letsencrypt.org/">Let's Encrypt</a>. As I was setting up the certificate though, I encountered a couple of articles warning the community about <a href="https://www.startssl.com/" rel="nofollow">StartSSL</a>. You can read more about it <a href="https://security.googleblog.com/2016/10/distrusting-wosign-and-startcom.html">here</a>.<br />
<br />
Since WoSign and StartCom are no longer trustworthy, I manually removed them from my browsers:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDb0BcBzABQ_8DNn45SI-n6rHNXYSBXPMEsQ8RflPtpWJET0juRwi1my2NeSh4Id9zcUGayhlR3sNc11YGKGZis0y1wNOtX2W4HgrI-CsnGnCwHkEPcg78laImshlIVZynHC7O/s1600/Screenshot-1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="456" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDb0BcBzABQ_8DNn45SI-n6rHNXYSBXPMEsQ8RflPtpWJET0juRwi1my2NeSh4Id9zcUGayhlR3sNc11YGKGZis0y1wNOtX2W4HgrI-CsnGnCwHkEPcg78laImshlIVZynHC7O/s640/Screenshot-1.png" width="640" /></a></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzJfwgSYcU2AE4B9oHT2jTcTSYAgPWZjMFlfpVgdypT1yGty8XV47yw3wrIvVqXNyCNXerZpomBKbOxoCmXTj_7Tuq6RlqR8xbpzxb2_roKOPY1shyphenhyphen8w3YlhSrd1gRTw-jWUbz/s1600/Screenshot-2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="457" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzJfwgSYcU2AE4B9oHT2jTcTSYAgPWZjMFlfpVgdypT1yGty8XV47yw3wrIvVqXNyCNXerZpomBKbOxoCmXTj_7Tuq6RlqR8xbpzxb2_roKOPY1shyphenhyphen8w3YlhSrd1gRTw-jWUbz/s640/Screenshot-2.png" width="640" /></a></div>
<br />
At least the chance of being snooped upon by Chinese state hackers is slightly lessened now.simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-38044318777694921812016-11-28T15:13:00.004+08:002016-11-28T17:57:34.004+08:00STARTTLS with Let's Encrypt and PowerMTA<span style="font-family: inherit;">This is an update to my previous post. I tried using the same certificate to let PowerMTA secure our e-mail server using STARTTLS, but unfortunately, Let's Encrypt is still untrusted by our PHP front-end because <a href="https://letsencrypt.org/">Let's Encrypt</a> is still a new certificate authority. The solution is to include IdentTrust's root certificate as it's already trusted by all clients.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">This setup is specific to a server with Apache and PowerMTA. You can probably make this work without a web server. Simply adjust the paths and other things as necessary.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">I'm pasting the previous steps in this post with some updates.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">First, log in as root. It's supposed to be a bad practice to work directly on root, but what the heck:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"># sudo su -</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Then, download GetSSL to your box:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"># mkdir -p /root/getssl /etc/letsencrypt</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"># cd /root/getssl</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"># curl -s https://raw.githubusercontent.com/srvrco/getssl/master/getssl > getssl</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"># chmod 700 getssl</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Then, generate the default config files:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"># ./getssl -c example.org </span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Making domain directory - /root/.getssl/example.org </span><br />
<span style="font-family: "courier new" , "courier" , monospace;">creating domain config file in /root/.getssl/example.org/getssl.cfg</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Edit the global config at: </span><span style="font-family: "courier new" , "courier" , monospace;">~/.getssl/getssl.cfg</span><span style="font-family: inherit;">. Pay particular attention to the followine line:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">RELOAD_CMD="service httpd reload"</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Depending on your system, it can be </span><span style="font-family: "courier new" , "courier" , monospace;">"service apache2 reload"</span><span style="font-family: inherit;"> or something similar. Update as necessary.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Uncomment the following line also:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">CA="https://acme-v01.api.letsencrypt.org"</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><span style="color: red;">WARNING</span>: The server has a limit of 5 certificates per 7 days, so use the staging server when experimenting. I hit the limit when writing this post. Lolz.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Edit the domain config at</span><span style="font-family: "courier new" , "courier" , monospace;"> ~/.getssl/example.org/getssl.cfg</span><span style="font-family: inherit;"> and insert the following lines:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">SANS=www.example.org</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">DOMAIN_KEY_CERT_LOCATION=/etc/letsencrypt/example.org.crt</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">SERVER_TYPE="https"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">CHECK_REMOTE="true"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">ACL=('/var/www/pub/.well-known/acme-challenge')</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">USE_SINGLE_ACL="true"</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Adjust the paths as necessary.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">The ACL variable above is the path for the Automatic Certificate Management Environment (ACME) challenge. It should point to a folder inside your web server's document root and should be accessible to the public via HTTP. The ACME challenge is used to prove that you control the web server.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Next, edit the Apache config and insert this in your SSL section:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Header always set Strict-Transport-Security "max-age=15768000"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">SSLProtocol all -SSLv2 -SSLv3</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">SSLHonorCipherOrder on</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">SSLCertificateFile /etc/letsencrypt/example.org.crt</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">SSLCertificateChainFile /etc/letsencrypt/le.crt</span><br />
<div>
<br /></div>
<div>
<span style="font-family: inherit;">Manually download Let's Encrypt's certificate:</span></div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /># curl -s https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem > /etc/letsencrypt/le.crt</span><br />
<div>
<br /></div>
<span style="font-family: inherit;">Next, we need to merge IdenTrust's root certificate with Let's Encrypt's. Open your browser and go to: <a href="https://www.identrust.com/certificates/trustid/root-download-x3.html">https://www.identrust.com/certificates/trustid/root-download-x3.html</a></span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Then, edit /etc/letsencrypt/le.crt and paste IdenTrust's cert at the bottom of the cert. le.crt should look like:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">-----BEGIN CERTIFICATE-----</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">-----END CERTIFICATE-----</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">-----BEGIN CERTIFICATE-----</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">-----END CERTIFICATE-----</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">For convenience, you can also paste the certificate above. Don't do this on sensitive websites though.</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<div>
<span style="font-family: inherit;">Then, execute GetSSL:</span></div>
<span style="font-family: inherit;"><br /></span><span style="font-family: "courier new" , "courier" , monospace;">./getssl -u -a</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: inherit;"></span>
<span style="font-family: inherit;">Verify if your certificates are OK:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"># openssl verify -CAfile /etc/letsencrypt/le.crt /etc/letsencrypt/example.org.crt</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">/etc/letsencrypt/example.org.crt: OK</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Restart Apache:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"># service httpd restart</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Stopping httpd: [ OK ]</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Starting httpd:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Finally, edit your PowerMTA config:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">smtp-server-tls-certificate /etc/letsencrypt/example.org.crt</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">smtp-server-tls-ca-file /etc/letsencrypt/le.crt</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"><source 0/0></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> allow-starttls yes</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> require-starttls-before-auth yes</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></source></span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Reload PowerMTA:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"># service pmta restart</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Stopping PowerMTA: [ OK ]</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Starting PowerMTA: [ OK ]</span><br />
<div>
<br /></div>
<span style="font-family: inherit;">Let's Encrypt's certificate expires every 90 days, so you need to update your crontab:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"># crontab -e</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Insert the following line in the crontab:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">0 0 1 * * /root/getssl/getssl -u -a</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Remember to protect your private keys and set the </span>appropriate<span style="font-family: inherit;"> permissions. Just leave a comment if I missed some steps.</span>simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com1tag:blogger.com,1999:blog-18519481.post-41106440512762032502016-11-24T14:31:00.004+08:002016-11-25T12:38:00.561+08:00Let's Encrypt on RHEL 6.8It's been almost 3 years since I've last posted in this blog. 2016 is almost over. I think I should make it a point to post regularly.<br />
<br />
Anyway, I'd like to share something interesting. There's a certificate authority called <a href="https://letsencrypt.org/">Let's Encrypt</a> that gives free certificates through an automated process. According to their website, they have a Python script called Certbot that fetches the certificates and updates your web server's configuration for you.<br />
<br />
Usually, I just use <a href="https://www.startssl.com/">StartSSL</a> for project prototypes or paid certificates on production websites, so I'm pretty late on discovering this.<br />
<br />
I'm currently tasked on setting up a server here in my work. The goal is to let others send information towards this server and automate the process of setting up PowerMTA configuration based on this information. The most straightforward and relatively secure way to do this is over HTTPS.<br />
<br />
Problem: The particular server that I'm working on runs on Red Hat Enterprise Linux 6.8, and Certbot needs updated Python packages to run. After a couple of attempts to solve this, I realized that it's a lost cause so I abandoned it and looked for a more lightweight solution with little dependencies. The best client that I've found so far is <a href="https://github.com/srvrco/getssl">GetSSL</a>. It's a straightforward bash script, and I like it!<br />
<br />
The server that I'm working on runs on Apache, but I can't find any step-by-step documentation on how to use GetSSL with Apache. Thus, I decided to make one.<br />
<br />
First, download GetSSL to your box:<br />
<br />
<pre style="background: #F7F7F7;"># curl --silent https://raw.githubusercontent.com/srvrco/getssl/master/getssl > getssl
# chmod 700 getssl
</pre>
<br />
Then, generate the default config files:<br />
<br />
<pre style="background: #F7F7F7;"># ./getssl -c example.org
Making domain directory - /root/.getssl/example.org
creating domain config file in /root/.getssl/example.org/getssl.cfg
</pre>
<br />
Edit the global config at: <span style="background-color: #f7f7f7; font-family: monospace;">~/.getssl/getssl.cfg</span>. Pay particular attention to the following line:<br />
<span style="background-color: #f7f7f7;"><br />RELOAD_CMD="service httpd reload"</span><br />
<br />
Depending on your system, it can be <span style="background: rgb(247 , 247 , 247); font-family: monospace;">service apache2 reload</span> or something similar. Update as necessary.<br />
<br />
Then, edit the domain config at: <span style="background-color: #f7f7f7; font-family: monospace;">~/.getssl/example.org/getssl.cfg</span> and insert the following lines:<br />
<br />
<pre style="background: #F7F7F7;"></pre>
<pre style="background: #F7F7F7;"></pre>
<pre style="background: #F7F7F7;">SANS=www.example.org
DOMAIN_CERT_LOCATION="/etc/httpd/getssl/example.org.crt"
DOMAIN_KEY_LOCATION="/etc/httpd/getssl/example.org.key"
DOMAIN_CHAIN_LOCATION="/etc/httpd/getssl/example.org-chain.crt"
CA_CERT_LOCATION="/etc/httpd/getssl/le.crt"
SERVER_TYPE="https"
CHECK_REMOTE="true"
ACL=('/var/www/.well-known/acme-challenge'
'/var/www/.well-known/acme-challenge')
</pre>
Adjust the paths as necessary.<br />
<br />
The ACL variable above is the path for the Automatic Certificate Management Environment (ACME) challenge. It should point to a folder inside your web server's document root and should be accessible to the public via HTTP. The ACME challenge is used to prove that you control the web server.<br />
<br />
Finally, edit the Apache config and insert this in your SSL section:<br />
<br />
<pre style="background: #F7F7F7;"></pre>
<pre style="background: #F7F7F7;">SSLCertificateFile /etc/httpd/getssl/example.org-chain.crt
SSLCertificateKeyFile /etc/httpd/getssl/example.org.key
SSLCACertificateFile /etc/httpd/getssl/le.crt
</pre>
Let's Encrypt's certificate expires every 90 days, so you need to update your crontab to automatically renew your certificate:<br />
<br />
<pre style="background: #F7F7F7;"># crontab -e
</pre>
<br />
Insert the following line in the crontab:<br />
<br />
<pre style="background: #F7F7F7;">0 0 * * * /path/to/getssl -u -a -q
</pre>
<br />
Save your changes and apply your changes:<br />
<br />
<pre style="background: #F7F7F7;">service httpd reload
</pre>
<br />
If everything is OK, go back to your global config at <span style="background-color: #f7f7f7; font-family: monospace;">~/.getssl/getssl.cfg</span> and uncomment the following line to use the non-staging server:<br />
<br />
<pre style="background: #F7F7F7;">CA="https://acme-v01.api.letsencrypt.org"
</pre>
<br />
Then, reload Apache again:<br />
<br />
<pre style="background: #F7F7F7;">service httpd reload
</pre>
<br />
That's it! Your web server is now running HTTPS. This won't really protect you from being particularly targeted by the NSA (there was a couple of state-sponsored attacks on certificate authorities), but the goal is to make broad mass-surveillance uneconomical. Just leave a comment if I missed some steps. Enjoy, and encrypt all the things!simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-60342260585640586592014-02-25T17:44:00.002+08:002014-02-25T17:45:52.025+08:00Aws\Ses\SesClient\sendRawEmail() needs base64Problem: You need to send raw e-mails to Amazon SES using the official PHP AWS-SDK, but Aws\Ses\SesClient\sendRawEmail() doesn't work even though your raw text is correct.<br />
<br />
Solution: Encode your data in base64. Had to check their sample Perl script just to discover this. :-/<br />
<br />
Working example:<br />
<pre><?php</pre>
<pre></pre>
<span style="font-family: monospace;"><span style="white-space: pre;">require 'AWSSDKforPHP/aws.phar';
use Aws\Ses\SesClient;
$client = SesClient::factory(Array(</span></span><br />
<pre> 'key => 'foobar',</pre>
<pre> 'secret' => 'foobarsecret',
'region' => 'us-east-1'
));
$data =<<EOD</pre>
<pre><eod from:="" hello="" info="" jobs-info.net="" obs="">From: "Example" <user@example.org>
To: "Simon Cornelius P. Umacob" <simoncpu@simoncpu.example.org><simon crowdmetric.com="">
hello world!
EOD;
$tmp = $client->sendRawEmail(Array(
'Source' => 'user@example.org',
'Destinations' => Array('simoncpu@simoncpu.example.org'),
'RawMessage' => Array(
<b>'Data' => base64_encode($data)</b>
)
));
var_dump($tmp);
?>
</simon></eod></pre>
simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-86251207071450481562012-02-21T03:45:00.003+08:002012-02-21T03:49:31.606+08:00Kohana nested SELECT statment<b>Problem</b><br />
You need to join two tables using a nested SELECT statement. However, Kohana's Database module doesn't really allow this query:<b> </b><br />
<pre>mysql> select forum_categories.*, (
-> select count(forum_data.id)
-> from forum_data
-> where forum_data.category_id = forum_categories.id and
-> forum_data.parent_id = '0'
-> ) as cat_count
-> from forum_categories;
+----+----------------+------------------------+-----------+
| id | title | description | cat_count |
+----+----------------+------------------------+-----------+
| 1 | Category One | category 1 description | 2 |
| 2 | Category Two | category 2 description | 2 |
| 3 | Category Three | category 3 description | 0 |
| 4 | Category Four | category 4 description | 0 |
| 5 | Category Five | category 5 description | 1 |
| 6 | Category Six | category 6 description | 0 |
+----+----------------+------------------------+-----------+
6 rows in set (0.01 sec)
</pre><br />
<b>Solution</b><br />
Rewrite your query as a left outer join (or inner join, depending on your case): <br />
<pre>mysql> select forum_categories.*, ifnull(forum_data.total, 0) as total
-> from forum_categories
-> left join (
-> select category_id, count(forum_data.id) as total
-> from forum_data where parent_id = '0'
-> group by category_id
-> ) as forum_data
-> on forum_data.category_id = forum_categories.id;
+----+----------------+------------------------+-------+
| id | title | description | total |
+----+----------------+------------------------+-------+
| 1 | Category One | category 1 description | 2 |
| 2 | Category Two | category 2 description | 2 |
| 3 | Category Three | category 3 description | 0 |
| 4 | Category Four | category 4 description | 0 |
| 5 | Category Five | category 5 description | 1 |
| 6 | Category Six | category 6 description | 0 |
+----+----------------+------------------------+-------+
6 rows in set (0.01 sec)
</pre><br />
Translating to Kohana, we now have:<br />
<br />
<pre>$subquery = DB::select('category_id', array('COUNT("forum_data.id")', 'total'))
->from('forum_data')
->where('parent_id', '=', '0')
->group_by('category_id');
$query = DB::select('forum_categories.*', array('IFNULL("forum_data.total", 0)', 'total'))
->from('forum_categories')
->join(array($subquery, 'forum_data'), 'LEFT')
->on('forum_data.category_id', '=', 'forum_categories.id')
->as_object()
->execute();
</pre><br />
For more info, check out the <a href="http://kohanaframework.org/3.2/guide/database/query/builder#subqueries">Subqueries section</a> of Kohana's Query Builder documentation.simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-62100915518981696202011-10-13T14:35:00.003+08:002011-10-13T15:44:33.477+08:00So long, farewell, and thanks for all the fish!<pre>#include <stdio.h>
int main(void)
{
printf("Goodbye world... :( \n");
return 70;
}
</pre>simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-22732357764745366692011-07-30T20:04:00.003+08:002011-08-14T17:03:19.232+08:00HOWTO Remove the annoying floating ads from Gmail's Preview Theme<b>Problem</b><br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirOXh4Fg33RObbWSTuy2bWkGIEzsUCPXEpZx11g-1J598RN3BkQHhd8TVnVWo-icDdbEhUZffawFkxlmZ_3clrWG58w33zYjVILB41NEiuD8USRSzbEJUQAnnpyT48hbes-sEC/s1600/Gmail+-+Google+Chrome.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="267" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirOXh4Fg33RObbWSTuy2bWkGIEzsUCPXEpZx11g-1J598RN3BkQHhd8TVnVWo-icDdbEhUZffawFkxlmZ_3clrWG58w33zYjVILB41NEiuD8USRSzbEJUQAnnpyT48hbes-sEC/s320/Gmail+-+Google+Chrome.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Gmail's Preview Theme without floating ads.</td></tr>
</tbody></table>You switched your Gmail theme, and you find the floating advertisement annoying.<br />
<br />
<b>Solution (for Chrome)</b><br />
Modify Gmail's stylesheet to prevent the ads from floating.<br />
<ol><li>Install the <a href="https://chrome.google.com/webstore/detail/plcnnpdmhobdfbponjpedobekiogmbco">Personalized Web</a> Chrome extension.</li>
<li>Go to Tools -> Extensions.</li>
<li>Find the Personalized Web section and click "Options".</li>
<li>Click "Add New Rule".</li>
<li>In the "Match URLs" textbox, input "mail.google.com".</li>
<li>In the "Add CSS" textbox, input:<code><br />
.mq {<br />
position: relative;<br />
bottom: 0px;<br />
left: 0px;<br />
margin: 0px;<br />
}<span class="Apple-style-span" style="font-family: 'Times New Roman';"><br />
</span></code></li>
<li><code>Click "Save".</code></li>
</ol><div>Reload Gmail to see the effect.<br />
<br />
<b>FAQ</b><br />
Q: Why not just disable the advertisement in the first place?<br />
A: Nothing in life is free; Google is providing us with this free service in exchange for displaying ads. It's a fair deal.<br />
<br />
Q: Does this work on Firefox?<br />
A: No, but you can just google for steps on using a custom CSS. You may use the same CSS from Step 6.</div>simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-24610166833660664702011-07-30T17:47:00.005+08:002011-08-14T17:06:17.839+08:00Required packages for building an Android OS on Ubuntu<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8KSjK1HTuesyZPnVASSYcIrmO326E6_RT2gylgc7a5Vi9bF16PqmK-276MQna_QZJA87Ld_IuCclnRu_QTF-rNl8vXIBlw0rz_pQnJ_hzXnFqBbQwhUOexYrb8YgTxcfhXoJz/s1600/android-logo-white.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8KSjK1HTuesyZPnVASSYcIrmO326E6_RT2gylgc7a5Vi9bF16PqmK-276MQna_QZJA87Ld_IuCclnRu_QTF-rNl8vXIBlw0rz_pQnJ_hzXnFqBbQwhUOexYrb8YgTxcfhXoJz/s200/android-logo-white.png" width="200" /></a></div><b>Problem</b><br />
You're trying to install the required packages for building a complete Android Operating System from scratch. Upon pasting the commands from the official documentation, you get the following error:<br />
<br />
<code> E: Couldn't find package lib32ncurses5-dev<br />
E: Couldn't find package lib32readline5-dev<br />
E: Couldn't find package lib32z-dev<br />
E: Couldn't find package mingw32<br />
</code><br />
<br />
<b>Solution</b><br />
The list of required packages for building an Android OS is wrong. The correct packages are:<br />
<code><br />
sudo apt-get install git-core gnupg flex bison gperf build-essential \<br />
zip curl zlib1g-dev libc6-dev lib64ncurses5-dev \<br />
x11proto-core-dev libx11-dev lib64readline5-dev lib64z-dev \<br />
libgl1-mesa-dev g++-multilib tofrodos<br />
</code><br />
<br />
Note: This has been tested on Ubuntu 10.04 (Lucid Lynx). YMMV.simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-42891242531353713412011-03-16T22:54:00.008+08:002012-10-18T19:13:11.614+08:00Downgrading from PHP 5.3 to 5.2 on Debian Squeeze<span style="font-weight: bold;">Problem</span><br />
<br />
You've installed PHP 5.3 on your Debian Squeeze system. Problem is, some web apps have trouble running on that version. You need to downgrade your system from 5.3 to 5.2.<br />
<br />
<span style="font-weight: bold;">Solution</span><br />
<br />
Remove the PHP 5.3 packages from your system:<br />
<blockquote style="font-family: courier new;">
sudo aptitude purge `dpkg -l | grep php| awk '{print $2}' |tr "\n" " "`</blockquote>
Clean the cache just to be sure:<br />
<blockquote style="font-family: courier new;">
rm -f /var/cache/apt/archives/php5*</blockquote>
Use Karmiс for PHP packages:<br />
<span style="font-family: courier new;"></span><br />
<blockquote>
<span style="font-family: courier new;">echo -e "Package: php5\nPin: release v=karmic\nPin-Priority: 991\n" | sudo tee /etc/apt/preferences.d/php > /dev/null</span><br />
<span style="font-family: courier new;">apt-cache search php5-|grep php5-|awk '{print "Package:", $1,"\nPin: release v=karmic\nPin-Priority: 991\n"}'|sudo tee /etc/apt/preferences.d/php > /dev/null</span></blockquote>
Add Ubuntu Karmic to source list:<br />
<span style="font-family: courier new;"></span><br />
<blockquote>
<span style="font-family: courier new;">cd /etc/apt/sources.list.d</span><br />
<span style="font-family: courier new;">sudo wget -O karmic.list "http://pastebin.com/download.php?i=q9ya307g"</span></blockquote>
<span style="color: red;">(Update, October 18, 2012: The sources list has been updated, because Ubuntu no longer supports Karmic. Please leave a comment if this still doesn't work.)</span><br /><br />
Update the package database:<br />
<blockquote>
<span style="font-family: courier new;">sudo apt-get update</span></blockquote>
<span style="font-family: courier new;"></span>If the command above produces this error:<br />
<span style="font-family: courier new;"></span><br />
<blockquote>
<span style="font-family: courier new;">W: GPG error: http://security.ubuntu.com karmic-security Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 40976EAF437D05B5</span><br />
<span style="font-family: courier new;">W: GPG error: http://archive.ubuntu.com karmic Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 40976EAF437D05B5</span><br />
<span style="font-family: courier new;">W: GPG error: http://archive.ubuntu.com karmic-updates Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 40976EAF437D05B5</span><span style="font-family: courier new;"><br /></span></blockquote>
Then import the required keys and add them to your list of trusted keys:<br />
<span style="font-family: courier new;"></span><br />
<blockquote>
<span style="font-family: courier new;">gpg --keyserver hkp://subkeys.pgp.net --recv-keys 40976EAF437D05B5</span><br />
<span style="font-family: courier new;">gpg --export --armor 437D05B5 | sudo apt-key add -</span><br />
<br />
<span style="font-family: courier new;">gpg --keyserver hkp://subkeys.pgp.net --recv-keys 40976EAF437D05B5</span><br />
<span style="font-family: courier new;">gpg --export --armor 40976EAF437D05B5 | sudo apt-key add -</span><br />
<br />
<span style="font-family: courier new;">gpg --keyserver hkp://subkeys.pgp.net --recv-keys 40976EAF437D05B5</span><br />
<span style="font-family: courier new;">gpg --export --armor 40976EAF437D05B5 | sudo apt-key add -</span></blockquote>
Finally, install PHP 5.2:<br />
<blockquote style="font-family: courier new;">
sudo apt-get install -t karmic php5-cli php5-cgi libapache2-mod-php5</blockquote>
UPDATE: Using the Ubuntu sources to download old packages works on Debian. So far, it works on my system, but you may need to use the Debian sources just in case.simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com14tag:blogger.com,1999:blog-18519481.post-43713556885234440632010-08-05T12:40:00.004+08:002010-08-05T12:52:11.274+08:00I hate Linux part 2OK, I was able to run <a href="http://www.sysresccd.org/">SystemRescueCd</a>'s USB creator by running it in Ubuntu Lucid. The USB stick booted, but when I ran GParted, it wasn't able to detect the RAID array.<br /><br /><span style="font-weight:bold;">Linux = FAIL.</span><br /><br />The point of this exercise was to run GParted in order to resize our FreeBSD's /usr partition to make way for ZFS. I allocated around ~400GB of space to that partition, but I realized that it's better to use that space for ZFS since I'll be using it for jails.<br /><br /><span style="font-weight:bold;">FreeBSD jails + ZFS = WIN.</span><br /><br />I'll just copy /usr to /var, resize it, and move back /usr to the new partition. I hope this works.<br /><br /><br />(this is probably GParted's fault, but I love to troll hahaha)simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com1tag:blogger.com,1999:blog-18519481.post-79783747339623363642010-08-03T00:51:00.005+08:002010-08-03T01:01:24.908+08:00I hate Linux<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDhi6jdgnPRzQ8rKGxtOV2azCrnR-oEZUiqC3ZBTFfJhnWnXUOzR21ADhzC3yy0T7ZcMJYM3I5VnujH-h9Wg5Y3xKkMQC5MO3gf5vF3ATAlSjiJUmkzEgt9PeM3xmndNg9Kofh/s800/tux_gletu_cipni.jpg"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer; width: 192px; height: 195px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDhi6jdgnPRzQ8rKGxtOV2azCrnR-oEZUiqC3ZBTFfJhnWnXUOzR21ADhzC3yy0T7ZcMJYM3I5VnujH-h9Wg5Y3xKkMQC5MO3gf5vF3ATAlSjiJUmkzEgt9PeM3xmndNg9Kofh/s800/tux_gletu_cipni.jpg" alt="" border="0" /></a><br />Yet another reason why I hate Linux:<br /><br /><a href="http://www.sysresccd.org/">SystemRescueCd</a>, a Linux distro designed for administering or repairing a system, doesn't boot and stops with the following error:<br /><br /><span style="font-family:courier new;">!! Cannot find device with /sysrcd.dat. Retrying...</span><br /><br />After a bit of googling, it turns out that this occurs because Linux can't detect the USB CD-ROM that was used to boot it. This use case isn't QA'd anymore. Apparently, the solution is to use a USB stick instead.<br /><br />But what happens if you use a USB stick? It doesn't work:<br /><br /><span style="font-family:courier new;">[root@soulfury mnt]# mount -o loop,exec ~simoncpu/Desktop/systemrescuecd-x86-1.5.8.iso cdrom</span><br /><span style="font-family:courier new;">[root@</span><span style="font-family:courier new;">soulfury</span><span style="font-family:courier new;"> mnt]# cd cdrom</span><br /><span style="font-family:courier new;">[root@</span><span style="font-family:courier new;">soulfury</span><span style="font-family:courier new;"> cdrom]# ls</span><br /><span style="font-family:courier new;">bootdisk bootprog isolinux ntpasswd sysrcd.dat sysrcd.md5 usb_inst usb_inst.sh usbstick.htm version</span><br /><span style="font-family:courier new;">[root@</span><span style="font-family:courier new;">soulfury</span><span style="font-family:courier new;"> cdrom]# bash usb_inst.sh </span><br /><span style="font-family:courier new;">No valid USB/Removable device has been detected on your system</span><br /><br />I checked the usb_inst.sh source code, and it makes some invalid assumptions on what a USB stick is. Blah. I need to sleep.simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-31586497615923367732010-07-26T23:20:00.006+08:002010-07-30T11:43:08.421+08:00FreeBSD: Unable to find device node for dev /dev/ar0s1b in /dev<span style="font-weight: bold;">Problem:</span><br />Sysinstall can't continue and stops with the error, "unable to find device node for dev /dev/ar0s1b in /dev".<br /><br /><span style="font-weight: bold;">Solution:</span><div>Remove the existing partition table by writing zeros at the start of the disk:<br /><pre>dd if=/dev/zero of=/dev/ar0 bs=64k count=1</pre>(replace as necessary, e.g., ad0)</div>simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-67978767916458943352010-07-17T00:08:00.001+08:002010-07-17T00:08:53.880+08:00Apache: Restricting folders by IP addresses<b>1st Problem:</b><br />
You are running an Apache server. You need to restrict access to certain folders by IP addresses.<br />
<br />
<b>Solution to 1st problem:</b><br />
Use .htacess to deny all access to that folder, then allow certain hosts/IP addresses.<br />
<br />
AuthName "simoncpu's dark secret"<br />
AuthType Basic<br />
<br />
<Limit GET POST><br />
order deny,allow<br />
deny from all<br />
allow from 10.0.0.<br />
allow from .example.org<br />
allow from this.is.an.example.invalid<br />
</Limit><br />
<br />
<b>2nd Problem:</b><br />
Your Apache is behind a proxy such as nginx. All requests to your Apache server appear to originate from your proxy.<br />
<br />
<b>Solution to 2nd problem:</b><br />
Install <a href="http://stderr.net/apache/rpaf/">mod_rpaf</a> and restart your web server. .htaccess will now correctly restrict access to your folders. Refer to its web page for installation instructions.simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-9025667200836293012010-07-04T21:33:00.006+08:002010-07-05T14:41:42.412+08:00Youtube comment hackAs of July 4, 2010, 9:30PM +8GMT, this Youtube comment hack works:<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><script>IF_HTML_FUNCTION?<h1><div style="background-color:#00000<wbr>0; position: absolute; top: 0; left: 0; font-size: 100px; color: red">simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu simoncpu <script></span><br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicDPrbIEGWaU5fG1DOSGlXuNRjwNz6A_hlbM1Hd3_xwwqpRASuaWmR73CY_Dj3QeBNFNh7cbjQ6Y0tdoUg_2t6eiQAFhCkTF7m9ef2qyA4kUCSiUFnXJmN2TdZ6C95asfcF-rO/s1600/Screenshot-YouTube+-+A+Deranged+INTP+-+Google+Chrome.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicDPrbIEGWaU5fG1DOSGlXuNRjwNz6A_hlbM1Hd3_xwwqpRASuaWmR73CY_Dj3QeBNFNh7cbjQ6Y0tdoUg_2t6eiQAFhCkTF7m9ef2qyA4kUCSiUFnXJmN2TdZ6C95asfcF-rO/s320/Screenshot-YouTube+-+A+Deranged+INTP+-+Google+Chrome.png" /></a></div><div class="separator" style="clear: both; text-align: center;"><br />
</div><div class="separator" style="clear: both; text-align: left;">Update: As of July 5, 2010, 2:40PM +8GMT, this hack no longer works.</div>simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-13528504027640746592010-04-27T21:01:00.001+08:002010-04-27T21:02:45.145+08:00Hello, Android!<a href="http://www.flickr.com/photos/simoncpu/4557137517/" title="Hello, Android! by simoncpu, on Flickr"><img alt="Hello, Android!" src="http://farm4.static.flickr.com/3631/4557137517_6b1a533569.jpg" width="375" /></a><br />
<br />
Yay, this is my first Android app! Java is really slow especially if you use Eclipse for development. I tried this on an amd64 machine with 2GB of RAM but this simple app still crawls to a halt. Anyway, I wish I can try this on a real phone.simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-44307543887983174502010-02-19T23:50:00.000+08:002010-02-19T23:50:45.230+08:00Moblin, Eeebuntu 4.0 beta, etcJust got back from Cebu GNU Linux Users Group meeting. I saw someone using Moblin. Three thumbs up! The UI is very intuitive, and will have no problems with mainstream customer adoption.<br />
<br />
Meanwhile, Eeebuntu 4.0 is not even ready for early adopters. The installation stopped somewhere at around 95% and complained about failed GRUB setup. I'm currently downloading Ubuntu 9.10 Netbook Remix because my Eee is left with a broken OS. Let's see how this goes.simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-14806450861133119932010-02-08T23:14:00.006+08:002010-03-30T16:12:16.827+08:00FreeBSD on EeeI've managed to install FreeBSD 8.0 on my Eee. I was planning to install NetBSD 5.0, but I found out that I had to convert the ISO to a memory stick image before I can dd it. For some reasons, <a href="http://unetbootin.sourceforge.net/">unetbootin</a> doesn't work with NetBSD's ISOs. There's a script that converts NetBSD ISOs to memstick images, but it seems that it only works on an existing *BSD system. I could have just SSH'd into a remote NetBSD system so that I can run the script there, but I realized that the output would be too big. SSH access is a privilege and not a right; abusing it with large file transfers is a big no-no.<br />
<br />
I tried to manually create an image using Mac OS X, but I couldn't initialize my USB stick's MBR (i.e., I couldn't boot from the USB). It says that it couldn't find "/usr/standalone/i386/boot0". I searched for it at the Mac OS X' original install discs, but I couldn't find it there either.<br />
<br />
It is possible to install NetBSD using Mac OS X, but I was like, FreeBSD is already available as a memstick image from its FTP site. Why not use it? Thus, I downloaded it, dd'd it to my USB stick, and installed it on my Eee. Everything was straightforward.<br />
<br />
Straightforward is boring. I'll try to recompile the kernel to see if I can further reduce its boot time (it boots less than 20sec from GRUB to login). I strongly suspect that I'll just reduce its space but not its boot time, but I'll never know until I try. I'll also try to install NetBSD via PXE, once I'm done playing with FreeBSD. I haven't tried doing it yet.simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0tag:blogger.com,1999:blog-18519481.post-3209110774515288762010-02-06T15:17:00.003+08:002010-02-06T16:02:49.226+08:00Eeexperiments<span style="font-weight: bold;">Moblin<br /><br /></span>I've finally got around installing Moblin. I've copied the image to a USB stick and attempted to boot the OS. The boot process didn't continue though. I got the following error:<pre>/dev/sda does not contain a rootfs<br />/dev/sdb does not contain a rootfs<br />/dev/sdc does not contain a rootfs<br />/dev/sdd does not contain a rootfs</pre>I might try tweaking the kernel configs some time to see if it would work. Meanwhile, I'll just install NetBSD.<br /><br /><span style="font-weight: bold;">Boot Booster<br /><br /></span>Eee's BootBooster reduced the boot time by skipping the pre-boot sequence. Eeebuntu <strike>wasted</strike> installed a swap partition when I first installed it, but I never used it because I know that swapping would increase the number of writes to my Eee's SSD. Increasing the number of writes would shorten its life span. Thus, I deleted my swap and installed an EFI partition in its place. Once the BIOS detected the new partition, the BootBooster option magically appeared.<br /><br /><span style="font-weight: bold;">coreboot</span><br /><br />A quick googling revealed that coreboot is unsupported on an Eee PC 701. I've also stumbled upon flashrom's source code (Linux bios flasher), and <a href="http://code.coreboot.org/p/flashrom/source/tree/HEAD/trunk/print.c">it explicitly </a><span class="str"><a href="http://code.coreboot.org/p/flashrom/source/tree/HEAD/trunk/print.c">stated that Eee PC 701 has been verified to be unsupported</a>. Oh well.</span>simoncpu-testhttp://www.blogger.com/profile/03046915034385308620noreply@blogger.com0