Reason Number One for Rav Yehudai Ga’on
The Mordechai explains that a Mumar loses his status as a brother, as we find in the context of Hilchot Ribbit (at least according to the Shulchan Aruch, Yoreh Dei’ah 159:2, wherein
Rav Yoseif Karo permits one to lend money with interest to a Mumar; the Rama recommends to be strict when possible), and thus the Ach Mumar does not have the Halachic status of a brother to be Zokeik LeYibum.
The Avnei Milu’im (Even Ha’Ezer 157), however, challenges this explanation. The definitions of the word “Ach,” “brother,” regarding Ribbit and Yibum are different. Concerning Ribbit, the term ‘brother’ refers to the collective brotherhood of the Jewish People, and so a person who does not behave like a Jew loses his status as a brother. Brotherhood in the context of Yibum, by contrast, is determined by the blood relationship that brothers share, and thus it is not related to whether the brother is a Mumar or an observant Jew.
Rashi firmly rejects Rav Yehudai Ga’on’s ostensible line of reasoning in our case of Ach Mumar, as well as in a case of Ribbit. He marshals an important phrase from Sanhedrin 44a, “Chata Yisrael Af Al Pi She-Chata Yisrael Hu,” “’Israel has sinned’- even though he sinned, he remains a Jew,” to mean that no amount of sin removes a Jew from Klal Yisrael. (It is important to note, on the other hand, that in its original context in Sefer Yehoshua, the phrase “Chata Yisrael” refers to Achan, who does not seem to have rejected Judaism in toto, but merely broken one Halachah.) In addition, Rashi does not see a difference when the brother completely abandoned Torah, since Yevamot 23a states that any brother is considered a Yavam, including a Mamzeir. If so, a Mumar should also Zokeik LeYibum. However, Teshuvot Tzemach Tzedek (number 312) argues that just as in the context of Hilchot Ribbit we essentially do not follow Rashi, so too in the context of Hilchot Yibum do not follow Rashi.
Evidence is cited by the Aruch HaShulchan against Rav Yehudai Ga’on from the fact that Eisav, who is defined as a Mumar (Kiddushin 17b), is considered a brother in the Torah. The Pasuk specifically states (Devarim 2:4), “Atem Ovrim BiGvul Acheichem Bnei Eisav,” “You are passing through the boundary of your brethren, the Children of Eisav,” and (Malachi 1:2) “HaLo Ach Eisav LeYa’akov,” “Is Eisav not a brother to Ya’akov?” However, Teshuvot Ya’avetz (number 30) responds by distinguishing between the narrative sections of the Torah, where brotherhood is defined in broad terms, and the halachic portions of the Torah, where a more formal definition of brotherhood applies. In the context of Yibum, one may argue a Mumar is not regarded as a brother, despite the Torah’s loose description of Eisav as an Ach.
On the other hand, we should note that the Chazon Ish (Yoreh Dei’ah 2:16 and 2:28) argues that non-observant Jews today are not Mumarim. This is because one is classified as a Mumar only after receiving Tochachah. Today, since we lack anyone who can administer proper, halachic Tochachah, non-observant Jews are regarded as “pre-Tochachah” and thus not classified as Mumarim.
Reason Number Two for Rav Yehudai Ga’on
The Ohr Zaru’a presents another reason for Rav Yehudai’s ruling. He notes the halachic concept of “Kol DeMekadesih ADa’ata DeRabbannan Mekadeish,” "Everyone marries according to the will of the Rabbis" (Gittin 33a). In other words, every marriage contains an implicit condition that Chazal must accept it. Accordingly, Chazal may veto a marriage if one contracts it in an unethical manner. Indeed, Chazal may effect a divorce even when the Get given by the husband is invalid on a biblical level. Tosafot (Ketubot 3a s.v. ADa’ata and Gittin 33a s.v. Kol) note that we explicitly authorize Chazal in this manner, as the groom states that he is marrying his wife “KeDat Moshe VeYisrael,”"According to the laws of Moses and Israel."
The Ohr Zarua seeks to apply the idea of Hafka’at Kidushin (annulment of a marriage) to the case of the Ach Mumar, since Chazal did not wish marriages to result in noxious situations such as a woman falling into Zika LeYibum to an Ach Mumar.
However, a major problem exists with implementing Hafka'at Kiddushin in the case of Ach Mumar. The Gemara records many objectionable marriages that are binding, indicating that the Rabbis do not annul them. For example, Chazal do not annul the marriage of a Kohein and a divorcee, although such a marriage is explicitly prohibited by the Torah (Vayikra 21:7). While criteria presumably exist to determine when marriages may be annulled, the Gemara never articulates them. The Rashba (Ketubot 3a) provides some insight:
One may ask why a woman should remain an Agunah if her husband was lost in a body of water without finite boundaries (Mayim She'ein Lahem Sof). Let the Rabbis annul the marriage! The answer is that the Rabbis do not annul a marriage unless the man has handed his wife at least a [Rabbinically acceptable] Get.
Elsewhere (Teshuvot 1:1185), the Rashba adds:
We do not say that whenever a husband entered a marriage in an unethical manner the rabbis annulled the marriage. Rather, we believe that only in those specific instances in which Chazal state that the marriage is annulled do we actually annul the marriage.
The Beit Yosef (Even Ha’Ezer 28) and Rama (Even Ha’Ezer 28:21) clearly affirm that we do not invoke Hafka'at Kiddushin when it is not specifically employed by the Gemara. The Rama writes:
A community that institutes a policy, accepted by the entire community, that anyone who marries in the absence of ten men will have his marriage invalidated must nevertheless be strict and require a get [in this circumstance].
Accordingly, since the idea of Hafka’at Kidushin does not appear in the Gemara in the context of an Ach Mumar, we may conclude that later authorities are not authorized by Halacha to by Mafki’a the Kidushin in such a case. However, the Ohr Zarua counters that since so many Ge’onim mention that Ach Mumar is not Zokeik L’Yibum, we can argue that the Ge’onim had the authority to apply Hafka’at Kidushin (or had a tradition from the time of the Gemara that Chazal made such a Takanah - we often assume that the Ge’onim possessed and applied oral traditions from Chazal).
The problem with this approach is the aforementioned Gemara (Yevamos 47b) which states that the Kiddushin of a Mumar count as Kiddushin. We thus see that Chazal do not apply Hafka’at Kiddushin in the case of a Mumar. We may answer that 1) according to the aforementioned opinion of Radach, the Gemara in Yevamot applies only to a Mumar who observes some Torah practices. 2) A Mumar is not Mekadeish Al Da’at Rabbanan, and thus Chazal cannot be Mafki’a his Kidushin if he was the original husband; however, in the case of an Ach Mumar, the Kiddushin was effected by an observant Jew, and thus the Kiddushin was ADa’ata DeRabanan and possibly annulled when the woman became Zekuka LeYibum to the Ach Mumar.
Reason Number Three for Rav Yehudai Ga’on
Ritva (Yevamot 47b) agrees with Rashi that a Mumar remains a Jew despite his apostasy, based on the Gemara (Yevamos 47b). However, he agrees with Rav Yehudai Ga’on that a Mumar is not Zokeik LeYibum. This is due to a specific issue regarding Yibum. The Torah states that the purpose of Yibum is “VeHayah HaBechor Asher Teileid Yakum Al Sheim Achiv HaMeit VeLo Yimacheh Shemo MiYisrael,” “And it shall be, the firstborn whom she bears shall stand on his [the Yavam’s] dead brother’s name, so that his name not be wiped out from Israel.” A Mumar’s name, however, is already destroyed in Yisrael. Hence, how could a Mumar possibly be Mekayeim Sheim Achiv B’Yisrael (uphold his brother’s name in Israel), when he himself is lacking a Sheim!?
One could easily challenge Ritva’s approach. After all, since when do we apply the Halachah based on the reason for the Mitzvah? The Gemara (Bava Metzia 115a) records a Tannaitic debate about this matter, and the Halachah follows the opinion that Lo Darshinan Ta’ama DiKra (the Halacha is not applied based on the reason for the Mitzvah). This seems to challenge the position of Ritva. However, Teshuvot Torat Chesed (Even Ha’Ezer 20) argues (based on Bava Metzia 115a) that if the reason for the prohibition is stated explicitly in the Pasuk, then all would agree that Darshinan Ta’ama DiKra, and so the Ach Mumar may be excluded from the Halachot of Yibum and Chalitzah.
Reason Number Four for Rav Yehudai Ga’on
Teshuvot Sho’eil UMeishiv (2:3:91) presents another explanation for Rav Yehudai Ga’on. Although the Mumar is regarded as a Jew, he is ineligible to perform Yibum. The Gemara (Yevamot 49b) cites Abba Sh’aul (whom Ashkenazic Jews follow), who states, “HaKoneis Et Yevimto LeShem Noi LeShum Ishut ULeShum Davar Acheir KeIlu Pogei’a Be’Ervah VeKarov Ani Be’Einai Lihiyot HaVelad Mamzeir,” “One who marries his sister-in-law [Yevama] because of beauty, because of marriage, or because of any other thing (except for fulfillment of the Mitzvah) touches on illicit relations, and I am close to saying that the child is illegitimate.” Thus, since the Mumar may be presumed not to have the intention LeSheim Mitzvah, he is ineligible for Yibum. Since the Mumar is ineligible for Yibum, he is ineligible for Chalitza, as the Gemara (Yevamot 3a) states, “Kol She’Ein Oleh LeYibum Eino OlehLeChalitzah,” “All who cannot perform Yibum cannot perform Chalitzah.” Thus, the Ach Mumar is not Zokeik L’Yibum.
One may respond that this is quite a novel assertion. Abba Sha’ul’s position is normally understood as a rabbinic stringency to avoid Yibum for fear of violating the prohibition of having marital relations with one’s brother’s wife. It is quite shocking to see Abba Shaul’s concern taken in a lenient direction to disqualify the Ach Mumar from Chalitza1.
1 The issue as to whether one’s intentions or actions define the character of the action is subject to a debate between Rabba and Rava that is recorded in Menachot 64a. If one believes like Rava that one’s intentions define his actions then perhaps we can understand Abba Shaul as articulating a Torah level concern that may be applied both in a stringent and lenient direction.