[Kune - Enhancement #512] (New) Page static generation via htmlunit perf issues
Issue #512 has been reported by Pablo Ojanguren. ---------------------------------------- Enhancement #512: Page static generation via htmlunit perf issues http://redmine.ourproject.org/issues/512 * Author: Pablo Ojanguren * Status: New * Priority: Normal * Assignee: Pablo Ojanguren * Category: Server side * Target version: Unplanned * Resolution: * Tags: ---------------------------------------- Generation of static html pages for crawler -using htmlunit- is provoking a lack of performance sometimes. We want to avoid this controlling how and when htmlunit processes are executed. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://redmine.ourproject.org/my/account
Issue #512 has been updated by Pablo Ojanguren. Htmlunit process is controlled from this servlet filter class <pre> cc.kune.core.server.searcheable.SearchEngineServletFilter </pre> Init method sets thread configuration: <pre> public void init(final FilterConfig filterConfig) throws ServletException { this.filterConfig = filterConfig; cache = new Cache(); executor = Executors.newFixedThreadPool(THREADS); } </pre> We have to define following issues to address a solution: * WHEN have the server to avoid launching htmlunit? current server load %, current mem usage %... * WHAT behavior will the server have in that case? response HTTP 404?, response HTTP 500? I think crawlers are very sensitive to this! ---------------------------------------- Enhancement #512: Page static generation via htmlunit perf issues http://redmine.ourproject.org/issues/512#change-737 * Author: Pablo Ojanguren * Status: New * Priority: Normal * Assignee: Pablo Ojanguren * Category: Server side * Target version: Unplanned * Resolution: * Tags: ---------------------------------------- Generation of static html pages for crawler -using htmlunit- is provoking a lack of performance sometimes. We want to avoid this controlling how and when htmlunit processes are executed. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://redmine.ourproject.org/my/account
Issue #512 has been updated by Vicente J. Ruiz Jurado. Other thing I was thinking (and trying without success), was to maintain open the htmlunit WebClient, and not to do a client.closeAllWindows(); only when servlet destroy. I was trying to cache and make the page request faster, but... maybe we have to try again. By the way, as we are using cloudflare CDN, I've added this rule: kune.cc/*escaped_fragment* Cache level: Aggressive caching ---------------------------------------- Enhancement #512: Page static generation via htmlunit perf issues http://redmine.ourproject.org/issues/512#change-738 * Author: Pablo Ojanguren * Status: New * Priority: Normal * Assignee: Pablo Ojanguren * Category: Server side * Target version: Unplanned * Resolution: * Tags: ---------------------------------------- Generation of static html pages for crawler -using htmlunit- is provoking a lack of performance sometimes. We want to avoid this controlling how and when htmlunit processes are executed. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://redmine.ourproject.org/my/account
Issue #512 has been updated by Vicente J. Ruiz Jurado. Parent task set to #286 ---------------------------------------- Enhancement #512: Page static generation via htmlunit perf issues http://redmine.ourproject.org/issues/512#change-739 * Author: Pablo Ojanguren * Status: New * Priority: Normal * Assignee: Pablo Ojanguren * Category: Server side * Target version: Unplanned * Resolution: * Tags: ---------------------------------------- Generation of static html pages for crawler -using htmlunit- is provoking a lack of performance sometimes. We want to avoid this controlling how and when htmlunit processes are executed. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://redmine.ourproject.org/my/account
Issue #512 has been updated by Vicente J. Ruiz Jurado. After our last conversation, and your work with #70 I just added some mbean methods, and also I have refactorized a little bit this servlet. In short (when this is installed in kune.cc) we can debug this. ---------------------------------------- Enhancement #512: Page static generation via htmlunit perf issues http://redmine.ourproject.org/issues/512#change-788 * Author: Pablo Ojanguren * Status: New * Priority: Normal * Assignee: Pablo Ojanguren * Category: Server side * Target version: Unplanned * Resolution: * Tags: ---------------------------------------- Generation of static html pages for crawler -using htmlunit- is provoking a lack of performance sometimes. We want to avoid this controlling how and when htmlunit processes are executed. -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://redmine.ourproject.org/my/account
participants (1)
-
Redmine Comunes