|Abstract: ||具通用性(generalizability)之研究成果可提升該研究的用途，讓所得的結果、理論、及預測應用到其他的母體(population)、地區、環境、和時間。通用性(也稱為外部效度, external validity)在某些類型的研究中已有深入的探討，諸如，實證(positivist)、闡釋(interpretive)、後實證(post-positivist)、批判理論(critical theory)、建構(constructivist)、及參與/合作(participatory/cooperative)等類型。這些類型的研究有廣泛的文獻提供指引和方法論(methodology)說明不同的研究設計如何減損或加強研究成果的通用性，例如，研究方法的文獻談到危害外部效度的因素，同時也指出如何設計實徵實驗、進行資料分析以避免這些因素。然而，資訊系統研究(Information Systems Research, ISR)可以是實證研究、闡釋研究、設計科學研究(Design Science Research, DSR)、或這些研究的綜合體的方式行。
資訊系統(IS)建築師(Architect)的產出物(artifact)包括程式語言、設計樣式(design pattern)、元件(component)、軟體架構(software architecture)、及框架(framework)。通常他們在設計及評估產出物時都會採用IS導向的DSR。雖然在ISR的文獻中常提及通用性，但大都是在承認該研究在這方面力有未逮，很少有人面對問題討論如何確保該研究的通用性。此現象可能肇因於大家對DSR和ISR的通用性瞭解太少、相關文獻也欠缺，於是如何評估通用性成了一個知識缺口(knowledge gap)。
本研究綜合三個研究方法論來探討ISR的通用性，包含德爾菲法(Delphi Method)、專家訪談(Expert Interview)、及親身觀察(Participation-Observation)。在德爾菲法訪談和專家訪談的過程中，有兩位大學教授參與，他們有IS界的豐富經驗而且曾設計過知識表達框架。筆者在業界任職軟體建築師，也有多年的經驗，在此研究中扮演親身觀察者的角色。德爾菲法的過程讓有關通用性共同且有力的觀點得以達成共識，其餘零星觀點照理會被捨棄。不過，德爾菲法有其限制，它會有專業偏見(specialization bias)，亦即某個專家憑其特定領域的專長而提供的觀點可能因其他外行人不瞭解而被忽視。由於本研究屬探索性，而且有鑑於德爾菲法的不足，這些零星觀點還是用專家訪談和親身觀察做進一步的檢視。如此一來，之前被忽視的零星觀點經過專家訪談、親身觀察、以及持續的文獻探討後，其重要性還有浮現的機會。
Generalizability of research findings increases the usefulness of the research study, allowing its results, theories, and predictions to hold true across multiple populations, geographic places, settings, and times. Generalizability, also referred to as external validity, has been thoroughly researched for certain types of research, namely positivist (e.g. natural science research) and interpretive (e.g. social science research) paradigms, as well as postpositivist, critical theory, constructivist, and participatory/cooperative paradigms. For these types of research, there exists extensive literature providing guidelines and methodologies for understanding how research design choices detract from or enhance generalizability of research findings. For example, research methodology literature describes factors that threaten external validity; literature is available to guide researchers to design empirical experiments and conduct data analyses to mitigate factors that threaten external validity. However, Information Systems Research (ISR) can be conducted as positivist, interpretive, or design science research (DSR) or a combination of all.
Information Systems Architects commonly conduct IS-oriented DSR as they design and evaluate artifacts, among which include programming languages, design patterns, components, software architectures, and frameworks. Although generalizability is often touched upon in ISR literature, it is usually only brought up as an admission to the limitations of various research studies and rarely confronted head-on with a solution to ensuring the generalizability of said studies. This seems to be due to a minimal understanding and consequently limited literature available on what generalizability actually is in both DSR and ISR, therefore creating a knowledge gap on how to evaluate it.
This study synthesizes three research methodologies to explore ISR Generalizability: the Delphi Method, Expert Interview, and Participant-Observation. Two university professors with extensive experience in the IS industry and who are directly involved in the design of a knowledge representation framework are included in a Delphi Method interview process and Expert Interviews. The author, also with extensive industry experience as a Software Architect, contributes as a participant-observer. The Delphi Method process allows common and strong viewpoints of generalizability to converge, leaving outliers that are normally discarded. However, since this is an exploratory study, and due to the limitation of the Delphi Method which overlooks specialization biases (in which one expert specializes in a particular aspect of the domain, therefore his or her contributions may not be understood or valued by the other experts who do not share the same specialization), these outliers are further examined using the Expert Interview method and in the participant-observation method. In this way, the importance of outliers previously mistakenly undervalued still have a chance to surface, based on expert interview results, participant observation, and continued literature review.
Before validating generalizability of any research undertaking, we need to understand what to assess and how to assess it; thus this study first develops the ISR Generalizability Framework. The constructs of Design, Scope, and Generalizability are defined and its relationships explained for the better understanding of generalizability in ISR. Scope varies from study to study, thus it becomes necessary to determine the domain-specific requirements for generalizability. Such requirements, such as statistical sampling limits, have been examined in detail for positivist and for other research paradigms, but not for ISR. Therefore this study also develops an ISR Generalizability Analysis Methodology to aid IS researchers in determining generalizability requirements for their specific ISR study. Because such requirements differ from study to study, it is important for the methodology to transcend the domain-specific or system-specific details of any particular research project. Instead, care is taken to maintain a generalized approach that is easily applied to any ISR study.
Generalizability in ISR both saves costs and improves reliability and maintainability. Previous literature offers little for the ISR and IS practitioner to fully understand, control-for, and analyze the generalizability of their ISR artifacts in a holistic manner. The results of this study draws from established literature in software reusability and IS design, plus adds a new dimension of domain-specific scoping requirements. The ISR Generalizability Framework and the ISR Generalizability Analysis Methodology developed in this study hopefully will aid future researchers and industry information architects in assessing and improving the generalizability and reusability of their ISR studies.
Keywords: generalizability, information systems research, design science research, research methodology, research paradigm, software design